exist-development

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[Thomas W. <tho...@gm...>]

I forgot to mention one side effect of this state - even the file is under
webapp\WEB-INF\data\fs, eXist can't find it and it throws an error:

514140 [qtp11513418-62] WARN org.eclipse.jetty.util.log - Nested in
javax.servlet.ServletException: An error occurred:
d:\eXist2\webapp\WEB-INF\data\fs\db\exe\res\exe-res\js\jquery.taconite.js
(The system cannot find the path specified):
java.io.FileNotFoundException:
d:\eXist2\webapp\WEB-INF\data\fs\db\exe\res\exe-res\js\jquery.taconite.js
(The system cannot find the path specified)

The way to recover is to delete both the collection that contains the binary
resources and the corresponding folder under  webapp\WEB-INF\data\fs.

This bug is a showstopper for me. I will really appreciate if somebody can
have a look at the issue.

Thomas

------

Thomas White

Mobile:+44 7711 922 966
Skype: thomaswhite
gTalk: thomas.0007
Linked-In:http://www.linkedin.com/in/thomaswhite0007
facebook: http://www.facebook.com/thomas.0007

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[James F. <jam...@gm...>]

one thing,

can you confirm if deletion from java client (or web client for that
matter) replicates this issue ?

also whats in the test.xql

J

On Tue, Mar 23, 2010 at 4:49 PM, Thomas White <tho...@gm...> wrote:
> I identified a problem and I have prepared a test file that reproduces the
> problem.
> When we move a collection to an existing collection with the same name the
> original folder corresponding to the original collection stays in the file
> system after the move. Then when we delete all collections, say using
> Oxygen, all folders in the fs stay there.
>
> To create the problem:
>
> 0. extract the zip into webaps/bres
> 1. execute http://localhost:8080/exist/bres/test.xql
> 2. check the fs : WEB-INF\data\fs\db\bres - there are two JS files in
> WEB-INF\data\fs\db\bres\moved-here\stored-coll\js
> 3. Delete /db/bres from Oxygen.
> 4. Check WEB-INF\data\fs\db\bres - all is fine.
>
> 5. execute http://localhost:8080/exist/bres/test.xql twice.
> 6.  check the fs : WEB-INF\data\fs\db\bres - not only
>  WEB-INF\data\fs\db\bres\moved-here\stored-coll\js  is there but
> WEB-INF\data\fs\db\bres\stored-coll where the directory was copied into the
> db initially is still there.
>
> 7. Delete /db/bres from Oxygen.
> 8. All directories under WEB-INF\data\fs\db\bres are still there. The
> original files are still there as well.
>
> the only way to fix it at the moment is to manually delete
> WEB-INF\data\fs\db\bres folder.
>
>
> Thomas
> ------
>
> Thomas White
>
> Mobile:+44 7711 922 966
> Skype: thomaswhite
> gTalk: thomas.0007
> Linked-In:http://www.linkedin.com/in/thomaswhite0007
> facebook: http://www.facebook.com/thomas.0007
>
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[Thomas W. <tho...@gm...>]

James,

Deleting from the the jave client has the same results. The files are
deleted but:
a) the folders in the fs are not deleted.
b) the originally imported files and directories in the fs are still there.
c) any subsequent import and move confuses eXist and the files are not
available for download.

In my original post there is an archive with all files necessary to
replicate the case.
Bellow is the content of test.xql.

Thomas


xquery version
"1.0" encoding "UTF-8";

declare default function namespace "http://www.w3.org/2005/xpath-functions";
declare namespace i18n="http://appache.org/cocoon/i18n/2.0";

declare boundary-space strip;
declare option exist:serialize "method=xml media-type=text/xml";

import module namespace xdb="http://exist-db.org/xquery/xmldb";

declare function local:webapp-home() {
let $home := system:get-exist-home()
let $pathSep := util:system-property("file.separator")
return
if (doc-available(concat("file:///", $home, "/webapp/index.xml"))) then
concat($home, $pathSep, "webapp")
else if(ends-with($home, "WEB-INF")) then
substring-before($home, "WEB-INF")
else
concat($home, $pathSep, "webapp")
};

declare function local:create-collection($parent as xs:string, $collection
as xs:string) {
let $r := xdb:create-collection($parent, $collection)
return
<li>Created collection {$r}</li>
};

declare function local:store-files($collection as xs:string, $home as
xs:string, $patterns as xs:string, $mimeType as xs:string?) as element()*
{
let $stored :=
if ($mimeType) then
xdb:store-files-from-pattern($collection, $home, $patterns, $mimeType, true
())
else
xdb:store-files-from-pattern($collection, $home, $patterns)
for $doc in $stored return
<li>Uploaded: {$doc}</li>
};


let $dir := concat( local:webapp-home(), util:system-property(
"file.separator"), 'bres' )
let $path := "/db/bres"
let $prep := (
xdb:create-collection("/db", "bres"),
xdb:create-collection($path, "moved-here")
)


return element imported-files {
element ul {
local:store-files($path, $dir, "**/*.js", "application/x-javascript"),
xdb:move( concat($path,'/stored-coll' ), concat($path,'/moved-here' ) )
}
}


------

Thomas White

Mobile:+44 7711 922 966
Skype: thomaswhite
gTalk: thomas.0007
Linked-In:http://www.linkedin.com/in/thomaswhite0007
facebook: http://www.facebook.com/thomas.0007



On 24 March 2010 12:41, James Fuller <jam...@gm...> wrote:

> one thing,
>
> can you confirm if deletion from java client (or web client for that
> matter) replicates this issue ?
>
> also whats in the test.xql
>
> J
>
> On Tue, Mar 23, 2010 at 4:49 PM, Thomas White <tho...@gm...>
> wrote:
> > I identified a problem and I have prepared a test file that reproduces
> the
> > problem.
> > When we move a collection to an existing collection with the same name
> the
> > original folder corresponding to the original collection stays in the
> file
> > system after the move. Then when we delete all collections, say using
> > Oxygen, all folders in the fs stay there.
> >
> > To create the problem:
> >
> > 0. extract the zip into webaps/bres
> > 1. execute http://localhost:8080/exist/bres/test.xql
> > 2. check the fs : WEB-INF\data\fs\db\bres - there are two JS files in
> > WEB-INF\data\fs\db\bres\moved-here\stored-coll\js
> > 3. Delete /db/bres from Oxygen.
> > 4. Check WEB-INF\data\fs\db\bres - all is fine.
> >
> > 5. execute http://localhost:8080/exist/bres/test.xql twice.
> > 6.  check the fs : WEB-INF\data\fs\db\bres - not only
> >  WEB-INF\data\fs\db\bres\moved-here\stored-coll\js  is there but
> > WEB-INF\data\fs\db\bres\stored-coll where the directory was copied into
> the
> > db initially is still there.
> >
> > 7. Delete /db/bres from Oxygen.
> > 8. All directories under WEB-INF\data\fs\db\bres are still there. The
> > original files are still there as well.
> >
> > the only way to fix it at the moment is to manually delete
> > WEB-INF\data\fs\db\bres folder.
> >
> >
> > Thomas
> > ------
> >
> > Thomas White
> >
> > Mobile:+44 7711 922 966
> > Skype: thomaswhite
> > gTalk: thomas.0007
> > Linked-In:http://www.linkedin.com/in/thomaswhite0007
> > facebook: http://www.facebook.com/thomas.0007
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Download Intel&#174; Parallel Studio Eval
> > Try the new software tools for yourself. Speed compiling, find bugs
> > proactively, and fine-tune applications for parallel performance.
> > See why Intel Parallel Studio got high marks during beta.
> > http://p.sf.net/sfu/intel-sw-dev
> > _______________________________________________
> > Exist-development mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-development
> >
> >
>

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[James F. <jam...@gm...>]

also, can you confirm if jquery.taconite.js an empty file ?

J

On Wed, Mar 24, 2010 at 9:56 AM, Thomas White <tho...@gm...> wrote:
> I forgot to mention one side effect of this state - even the file is under
> webapp\WEB-INF\data\fs, eXist can't find it and it throws an error:
>
> 514140 [qtp11513418-62] WARN org.eclipse.jetty.util.log - Nested in
> javax.servlet.ServletException: An error occurred:
> d:\eXist2\webapp\WEB-INF\data\fs\db\exe\res\exe-res\js\jquery.taconite.js
> (The system cannot find the path specified):
> java.io.FileNotFoundException:
> d:\eXist2\webapp\WEB-INF\data\fs\db\exe\res\exe-res\js\jquery.taconite.js
> (The system cannot find the path specified)
>
> The way to recover is to delete both the collection that contains the binary
> resources and the corresponding folder under  webapp\WEB-INF\data\fs.
>
> This bug is a showstopper for me. I will really appreciate if somebody can
> have a look at the issue.
>
> Thomas
> ------
>
> Thomas White
>
> Mobile:+44 7711 922 966
> Skype: thomaswhite
> gTalk: thomas.0007
> Linked-In:http://www.linkedin.com/in/thomaswhite0007
> facebook: http://www.facebook.com/thomas.0007
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[Thomas W. <tho...@gm...>]

James,

This JavaScript file is not empty. This is the result of running the import
of the files more then once.
If I delete the collection  AND the fs directory and then import the
JavaScript files - everything works fine. Until the second run of the
import.

BTW the error message I posted is not from the example it is from the
framework I am working on, but essentially it is the same thing

Thomas


------

Thomas White

Mobile:+44 7711 922 966
Skype: thomaswhite
gTalk: thomas.0007
Linked-In:http://www.linkedin.com/in/thomaswhite0007
facebook: http://www.facebook.com/thomas.0007



On 24 March 2010 12:42, James Fuller <jam...@gm...> wrote:

> also, can you confirm if jquery.taconite.js an empty file ?
>
> J
>
> On Wed, Mar 24, 2010 at 9:56 AM, Thomas White <tho...@gm...>
> wrote:
> > I forgot to mention one side effect of this state - even the file
> is under
> > webapp\WEB-INF\data\fs, eXist can't find it and it throws an error:
> >
> > 514140 [qtp11513418-62] WARN org.eclipse.jetty.util.log - Nested in
> > javax.servlet.ServletException: An error occurred:
> > d:\eXist2\webapp\WEB-INF\data\fs\db\exe\res\exe-res\js\jquery.taconite.js
> > (The system cannot find the path specified):
> > java.io.FileNotFoundException:
> > d:\eXist2\webapp\WEB-INF\data\fs\db\exe\res\exe-res\js\jquery.taconite.js
> > (The system cannot find the path specified)
> >
> > The way to recover is to delete both the collection that contains the
> binary
> > resources and the corresponding folder under  webapp\WEB-INF\data\fs.
> >
> > This bug is a showstopper for me. I will really appreciate if somebody
> can
> > have a look at the issue.
> >
> > Thomas
> > ------
> >
> > Thomas White
> >
> > Mobile:+44 7711 922 966
> > Skype: thomaswhite
> > gTalk: thomas.0007
> > Linked-In:http://www.linkedin.com/in/thomaswhite0007
> > facebook: http://www.facebook.com/thomas.0007
> >
> >
>  >
> ------------------------------------------------------------------------------
> > Download Intel&#174; Parallel Studio Eval
> > Try the new software tools for yourself. Speed compiling, find bugs
> > proactively, and fine-tune applications for parallel performance.
> > See why Intel Parallel Studio got high marks during beta.
> > http://p.sf.net/sfu/intel-sw-dev
> > _______________________________________________
> > Exist-development mailing list
> > Exi...@li...
> > https://lists.sourceforge.net/lists/listinfo/exist-development
> >
> >
>

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[Adam R. <ad...@ex...>]

Thomas,

Out of interest where is your EXIST_HOME located, I know on Windows
there are issues with the Sun JDK and operations on files under
specific folders of the USERPROFILE folders.

Thanks Adam.

On 23 March 2010 15:49, Thomas White <tho...@gm...> wrote:
> I identified a problem and I have prepared a test file that reproduces the
> problem.
> When we move a collection to an existing collection with the same name the
> original folder corresponding to the original collection stays in the file
> system after the move. Then when we delete all collections, say using
> Oxygen, all folders in the fs stay there.
>
> To create the problem:
>
> 0. extract the zip into webaps/bres
> 1. execute http://localhost:8080/exist/bres/test.xql
> 2. check the fs : WEB-INF\data\fs\db\bres - there are two JS files in
> WEB-INF\data\fs\db\bres\moved-here\stored-coll\js
> 3. Delete /db/bres from Oxygen.
> 4. Check WEB-INF\data\fs\db\bres - all is fine.
>
> 5. execute http://localhost:8080/exist/bres/test.xql twice.
> 6.  check the fs : WEB-INF\data\fs\db\bres - not only
>  WEB-INF\data\fs\db\bres\moved-here\stored-coll\js  is there but
> WEB-INF\data\fs\db\bres\stored-coll where the directory was copied into the
> db initially is still there.
>
> 7. Delete /db/bres from Oxygen.
> 8. All directories under WEB-INF\data\fs\db\bres are still there. The
> original files are still there as well.
>
> the only way to fix it at the moment is to manually delete
> WEB-INF\data\fs\db\bres folder.
>
>
> Thomas
> ------
>
> Thomas White
>
> Mobile:+44 7711 922 966
> Skype: thomaswhite
> gTalk: thomas.0007
> Linked-In:http://www.linkedin.com/in/thomaswhite0007
> facebook: http://www.facebook.com/thomas.0007
>
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
>



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[Thomas W. <tho...@gm...>]

EXIST_HOME  is  D:\eXist2
JAVA SDK HOME C:\Java\jdk1.6.0_16

Has anyone been able to reproduce this bug?

Thomas

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[Thomas W. <tho...@gm...>]

I was wandering if anybody can have a look at this issue.

It is pretty serious to me because after a second file import eXist goes out
of sync between the binary resource listed internally and the files that
actually are stored in the file system at the location they are expected.
And second, the fact that files stay in the file system after collections
deletions is a security issue as well.

I really hope somebody will be able to go to the bottom of this problem.
Thomas

------

Thomas White

Mobile:+44 7711 922 966
Skype: thomaswhite
gTalk: thomas.0007
Linked-In:http://www.linkedin.com/in/thomaswhite0007
facebook: http://www.facebook.com/thomas.0007



On 25 March 2010 17:28, Thomas White <tho...@gm...> wrote:

> EXIST_HOME  is  D:\eXist2
> JAVA SDK HOME C:\Java\jdk1.6.0_16
>
> Has anyone been able to reproduce this bug?
>
> Thomas
>

Re: [Exist-development] Binary resources not deleted when a collection is moved into existing collection with the same name.

[José M. F. G. <jm...@us...>]

Hi Thomas,
	I have taken a look at the bug you have reported, and following the codepaths the reason is that in NativeBroker, in moveBinaryFork, there is no check about the (lack of existence of) destination at binary resources level, so the move operation fails when there is already a binary resource with the same path. I have created a fix (see attached source patch), and I would like you to test it before I commit the changes to eXist trunk.

	Best Regards,
		José María

On 03/31/10 15:45, Thomas White wrote:
> I was wandering if anybody can have a look at this issue.
> It is pretty serious to me because after a second file import eXist goes
> out of sync between the binary resource listed internally and the files
> that actually are stored in the file system at the location they are
> expected. And second, the fact that files stay in the file system
> after collections deletions is a security issue as well.
> I really hope somebody will be able to go to the bottom of this problem.
> Thomas
>
> ------
>
> Thomas White
>
> Mobile:+44 7711 922 966
> Skype: thomaswhite
> gTalk: thomas.0007
> Linked-In:http://www.linkedin.com/in/thomaswhite0007
> facebook: http://www.facebook.com/thomas.0007
>
>
>
> On 25 March 2010 17:28, Thomas White <tho...@gm...
> <mailto:tho...@gm...>> wrote:
>
>     EXIST_HOME  is  D:\eXist2
>     JAVA SDK HOME C:\Java\jdk1.6.0_16
>     Has anyone been able to reproduce this bug?
>     Thomas
>
>
>
>
> ------------------------------------------------------------------------------
> Download Intel&#174; Parallel Studio Eval
> Try the new software tools for yourself. Speed compiling, find bugs
> proactively, and fine-tune applications for parallel performance.
> See why Intel Parallel Studio got high marks during beta.
> http://p.sf.net/sfu/intel-sw-dev
>
>
>
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development

-- 
"La violencia es el último recurso del incompetente"
	- Salvor Hardin en "La Fundación" de Isaac Asimov
"Premature optimization is the root of all evil." - Donald Knuth

José María Fernández González
e-mail: jos...@gm...