exist-development — eXist Developer's List

Re: [Exist-development] New eXist admin webapp - features list

[Adam R. <ad...@ex...>]

Indeed all configuration needs to become dynamic, and perhaps even
stored in the db with sane defaults if the configuration is missing.

2009/11/11 Dmitriy Shabanov <sha...@gm...>:
> On Wed, 2009-11-11 at 10:49 +0000, Thomas White wrote:
>>   A.  Server Configuration:
>>         1) have all config files available to change/set via web
>> interface: activate/deactivate modules; memory settings etc.
>>         2) Clone eXist configuration with save/restore of all
>> settings, transferable to other server.
>>         3) Set directories: data, backup, fs
>>         4) Indexes, triggers
>>         5) URL rewriting
>
> I have some configuration refactoring ideas, so it can be my part.
>
> --
> Cheers,
>
> Dmitriy Shabanov
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>



-- 
Adam Retter

eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb

[Exist-development] DOMaTeXist & refactoring

[Dmitriy S. <sha...@gm...>]

Hi every one,

The 1.4 is out! 1.6 on a way! (Le Roi est mort, vive le Roi!) ;) 

next I want to implement my old dream: one DOM-like (DOMaTeXist)
 top API for memory & stored.

I do understand that is very difficult to optimize alligator & hen, but
some sort of it should be possible. Any way, low API can be use to have
special magic.

At same time it will allow to have 
 - shared memory structure;
 - same API for any further implementation (on fly configuration as
example)
 - ... something that I can't remember, but very important staff ;)

The refactoring plan:
 - apply rule: clear name for interface, Impl suffix for class
 - move memtree to org.exist.dom & rename to memory
 - create org.exist.dom.internal & move related classes (stored nodes)
there
 - by SMALL steps create DOMaTeXist interface + deprecate (add) methods
that have same semantic, but different names at memory & stored

********************************************************** configuration

 - apply "configuration by annotations" 
( http://exist.svn.sourceforge.net/viewvc/exist/branches/shabanovd/config/src/org/exist/util/config/ )


*********************************************************** interpreter

 - add org.exist.interpreter & create interfaces (Context,
Interpreter ...) that should be a link between xquery & xsl engine
 - redesign xquery engine to split compiled tree & evaluation, after
demonstrating that technics on xslt

That's all for now, rest shouldn't touch core part. Well, not all, in
December I'll discuss phantom node integration. That will change a
little storage engine. (storage partitioning will require similar
changes any way)

That's my next 6 months plan ;) (svn, xslt, phNodes & more are not in
count)

If somebody have something more to change, please, share.

-- 
Cheers,

Dmitriy Shabanov

PS I did check my daily logs, still no plans to destroy eXist :-)

[Exist-development] Important notice: changes in subversion

[Dannes W. <di...@ex...>]

All,

I moved things a bit around in the subversion repository, to make it more consistent: The details:

in https://exist.svn.sourceforge.net/svnroot/exist i have created a 'stable' folder, containing all stable releases of eXist-db:

eXist-1.0.x
eXist-1.2.x
eXist-1.4.x

The latter is a copy of the current trunk, and will be the base for future versions like 1.4.1 etc. Note that if you have file checked out, you need to 'svn switch' here.

From now on 'trunk/eXist' is exist 1.5... unstable :-)


Kind regards

Dannes

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] Benchmarking eXist

[Dmitriy S. <sha...@gm...>]

On Thu, 2009-11-05 at 12:32 +0100, Jan Fabry wrote:

>   - How is testing currently a part of the eXist development process?  
> A page from the developer manual [ http://exist.sourceforge.net/developper.html 

better to use http://exist-db.org 

Take a look on org.exist.performance under test, but they are more
concurrent issue checker.

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] Benchmarking eXist

[Leif-Jöran O. <lj...@ex...>]

Den 2009-11-05 12:32, Jan Fabry skrev:
> Hello,
>
> For my bachelor's thesis, I am looking into the performance of XQuery
> engines. I see that eXist has some performance tests in the test
> suite, but I don't know whether they are still maintained or used as
> guides for further development. I would like to contribute something
> here, and was thinking of somehow combining the XCheck and TPoX
> benchmark frameworks to create a tool that would be useful to both
> researchers and developers.
>
> I would like to gather your input to know which features would be
> useful. If you could answer the following two questions, I would be
> very grateful:
>
>    - How is testing currently a part of the eXist development process?
> A page from the developer manual [ http://exist.sourceforge.net/developper.html
>    ] seems to be offline (but the Google cache is still available), but
> I don't know whether this is still current? Do you only use unit
> tests, or are the longer performance tests also used? (data.exist-
> db.org also seemed to be unreachable, so I could not download jgoethe-
> tei.zip for example)

Hi Jan,
The performance tests are maintained and augmented at occasions. Lastly 
we added authentic data for a EU project we are involved in.
We should probably move the pointer for data.exist-db.org to another 
host. Thanks for pointing this out.
We also use the xqts in conjunction with the unit and performance tests. 
The xqts tests are run with a larger time cycle frame than the unit 
tests on our build server though.

The utilisation graphs for the different build clients can in some way 
also be used to to see how the performance is affected by every build.

Cheers,
Leif-Jöran

Re: [Exist-development] New eXist admin webapp - features list

[Dmitriy S. <sha...@gm...>]

On Wed, 2009-11-11 at 10:49 +0000, Thomas White wrote:
>   A.  Server Configuration:  
>         1) have all config files available to change/set via web
> interface: activate/deactivate modules; memory settings etc.
>         2) Clone eXist configuration with save/restore of all
> settings, transferable to other server.
>         3) Set directories: data, backup, fs 
>         4) Indexes, triggers
>         5) URL rewriting

I have some configuration refactoring ideas, so it can be my part.

-- 
Cheers,

Dmitriy Shabanov

[Exist-development] New eXist admin webapp - features list

[Thomas W. <tho...@gm...>]

Dear all,

There are always more then one way to do one job - with XForms or with XSLT
it is a question of personal preferences. We can have more them one
implementation and people will gravitate to what they prefer.

I do not think it s the technology that defines the value of an application.
It is how well it does the main functions, how easy it is to  maintain and
extend and and new features etc.

Let us try to define what could be the best possible set of features and
then start implementing them following the principle of 20:80 - start from
these 20% features that are used 80% of the time.

I think we can have the following main areas:
*  A.  Server Configuration:*
        1) have all config files available to change/set via web
interface: activate/deactivate modules; memory settings etc.
        2) Clone eXist configuration with save/restore of all settings,
transferable to other server.
        3) Set directories: data, backup, fs
        4) Indexes, triggers
        5) URL rewriting

* B. Server information (view only):*
        1) Status,
        2) statistics
        3) performance monitor.
        4) profiling

* C. Job management*
      1) Currently running jobs
      2) Scheduled jobs
      3) Kill jobs, change job priority

* D. Admin actions:*
       1) Shutdown, restart
       2) Backup/Restore
       3) Run build and restart
       4) Reindex

*E. Security, User and Group Management*
      1) default security settings for collection and user
      2)  Users: single and bulk operations
      3) Groups
      4) Bulk(recursive) security operations

*F. Data: Collections and Resources*
     1) browse collections/resources:
          1.a. GIU with windows explorer style interface - folder tree on
the left, content on the right.
          1.b. Fast content browser where we see only the content of the
current entity and a cookie bread crumble at the top, for back and up
navigation. Clicking on a collection name or a resource will display its
contents. Every collection will have in brackets after its name {number or
sub collections}/{number of resources}. A resource will have its size after
its name. We can have views that present the content in a content specific
way like thumbnails if the collection contains images.
          1.c. Special attention should be paid to dealing with large number
of entities. We can have some way to summarize the content. We need
pagination, sorting and filtering.
     2) Security Operations on collections and resources: permissions,
groups, ownership
           2.a. single selected object
           2.b. multiple operation on selection by: checkbox, name, date,
mime type or other filtering criteria.
           2.c. recursive operations
     3) content operation
          3.a. Import: file, directory, directory with sub-folders
          3.b. Export: file, collection, collection with sub-collections


Have I missed something?

Thomas


------

Thomas White

Mobile:+44 7711 922 966
Skype: thomaswhite
gTalk: thomas.0007
Linked-In:http://www.linkedin.com/in/thomaswhite0007
facebook: http://www.facebook.com/thomas.0007

Re: [Exist-development] [Exist-open] Porting XForms applications to use XSLTForms

[Wolfgang M. <wol...@ex...>]

> Now that we are talking about the redesign of the admin website, we should probably create a new discussion in exist-development.

Thanks, Loren, you are right. We should definitely try to move more
discussions into the development list.

Wolfgang

Re: [Exist-development] split logs

[Dmitriy S. <sha...@gm...>]

On Sun, 2009-11-08 at 22:00 +0100, Dannes Wessels wrote:
> I think that if you want to craw through the logs of a complex
> product  
> as exist, you need some tooling. Log4j has chainsaw which is very  
> usable.....

As I can understand you don't use any tool to analyze logs. 
The main point: we need log analyzer & with it, split log or not will
mean nothing ;) Well, there was the plan to have web interface for logs.

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] split logs

[Dannes W. <di...@ex...>]

On 8 Nov 2009, at 8:12 , Dmitriy Shabanov wrote:

> Analyzing the exist.log very difficult, because all logs (almost all)
> there.
>
> Can we split them: webdav, rest, xquery & etc ? per threads (well,  
> that
> is bad, but do anybody know any tool to analyze logs)?


for webdav and validation we had for a long time a separate log file.  
Although it really made sense, it was very very difficult to find  
relations between entries in exist.log and webdav.log/validation.log ;  
very often i missed some essential relations.

Therefore, IMO it is a bad idea  to have the log split into separate  
files.

I think that if you want to craw through the logs of a complex product  
as exist, you need some tooling. Log4j has chainsaw which is very  
usable.....

In an other context I have to deal with logging of complex software as  
well. Multi threads, multi processor boards. In that context having  
one file is essential.

-1 for me.

(well you are free to change the log files locally :-) loren does that  
as well i think)


Kind regards

Dannes

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

[Exist-development] split logs

[Dmitriy S. <sha...@gm...>]

Good day,

Analyzing the exist.log very difficult, because all logs (almost all)
there. 

Can we split them: webdav, rest, xquery & etc ? per threads (well, that
is bad, but do anybody know any tool to analyze logs)?

What it give?
1. More similar information near.
2. Bigger log period. For now the logs have only about 3-5 hours of
activity in debug mode (on my server ;).  

Bad:
1. Need to analyze several logs as one. It can be solve by analyzer. 

-- 
Cheers,

Dmitriy Shabanov

[Exist-development] Benchmarking eXist

[Jan F. <jan...@mo...>]

Hello,

For my bachelor's thesis, I am looking into the performance of XQuery  
engines. I see that eXist has some performance tests in the test  
suite, but I don't know whether they are still maintained or used as  
guides for further development. I would like to contribute something  
here, and was thinking of somehow combining the XCheck and TPoX  
benchmark frameworks to create a tool that would be useful to both  
researchers and developers.

I would like to gather your input to know which features would be  
useful. If you could answer the following two questions, I would be  
very grateful:

  - How is testing currently a part of the eXist development process?  
A page from the developer manual [ http://exist.sourceforge.net/developper.html 
  ] seems to be offline (but the Google cache is still available), but  
I don't know whether this is still current? Do you only use unit  
tests, or are the longer performance tests also used? (data.exist- 
db.org also seemed to be unreachable, so I could not download jgoethe- 
tei.zip for example)
  - Do you have any ideas for performance testing that could not be  
realized yet?

If this information is already available somewhere, please excuse me  
for not finding it yet.

Thank you for your time,

Jan Fabry

[Exist-development] XQueryURLRewrite is not honoring "xquery" parameter as before

[José M. F. G. <jm...@us...>]

Hi everybody,
	meanwhile I was upgrading the eXist sources for an internal project which uses XQueryURLRewrite I have just realized that "xquery" configuration parameter is not taken into account. For that project I have a custom controller with a custom name, stored in the database, and now the custom controller is not being used because it is not called "controller.xql".

	I'm going to try fixing it, so if "xquery" configuration parameter is set to a document instead of a collection, the custom controller is taken into account.

	Best Regards,
		José María
-- 
"La violencia es el último recurso del incompetente"
	- Salvor Hardin en "La Fundación" de Isaac Asimov
"Premature optimization is the root of all evil." - Donald Knuth

José María Fernández González
e-mail: jm...@us...

Re: [Exist-development] Pluggable Authentication Architecture

[Dmitriy S. <sha...@gm...>]

That is very useful: http://wiki.openid.net/Relying-Party-Best-Practices

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] Pluggable Authentication Architecture

[Dmitriy S. <sha...@gm...>]

Is it possible to authenticate user without IdP redirects?

Of course, I can parse IdP response & apply username+password, but not
for all.

Is there any "standard" way?

-- 
Cheers,

Dmitriy Shabanov

[Exist-development] recovery proccess

[Dmitriy S. <sha...@gm...>]

Hi all,

The recovery process need some more attention. For now, no way to
reproduce recovery fail, because data files are changed by recovering &
nothing to analyze at the end.

I want to change it:

1. As configurable option. Copy data file before recovering. That will
leave data as it is & information to analyze.

2. Again, as configurable option. If recovering failed, delete data (if
option 1 disable copy&delete?) -> run fresh eXist -> restore from last
backup

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] Pluggable Authentication Architecture

[Dmitriy S. <sha...@gm...>]

Branched:

http://exist.svn.sourceforge.net/exist/?rev=10311&view=rev

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] Developing OpenID for eXist

[Loren C. <lor...@gm...>]

Please take a look at the following before the call:

http://en.wikipedia.org/wiki/AAA_protocol

https://support.syntactica.com/groups/existsecurity/

https://support.syntactica.com/groups/existsecurity/wiki/67cf3/Pluggable_Authentication_Architecture.html

https://support.syntactica.com/groups/existsecurity/wiki/fa687/Authentication_Entries_for_eXist.html

https://support.syntactica.com/groups/existsecurity/wiki/04f3c/Steps_to_implement_OpenID_in_eXist.html

https://support.syntactica.com/groups/existsecurity/wiki/dc956/Authorization.html

Once these wiki entries are completed, I intent on updating the  
security section of the documentation with them.  I will be populating  
the main wiki page with more of the Wikipedia on the AAA protocol.

Loren

Re: [Exist-development] admin menu

[Dmitriy S. <sha...@gm...>]

On Sat, 2009-10-31 at 22:23 +0100, Wolfgang wrote:
> > probably we should just restyle the whole admin interface with some  
> > yui (yahoo ui) ajax stuff. Some pages are already coded with this.
> 
> I planned to extend my small yahoo ui xquery module (yahoo-ui.xql) to 
> support menubars, so we could use one hierarchical menu bar on the top 
> of the page.
> 
> > otherwise.... xforms/serverside could do the trick?
> 
> I also added xforms to the admin webapp today (see the "Install Tools" 
> menu item). But I guess implementing the entire webapp in XForms would 
> be a bit too much. Anyway, we can mix both, XForms for the more complex 
> user input forms, yahoo ui for the general interface.

Well, if the plans are so big can we design UI top API? (content ... ,
but I didn't say that ;) And use it over all site.

1. Menu - yui
2. Forms - xforms
3. Translations - that is simple ;)
4. Themes - xquery + xsl, Who can give ideas how it should work?
5. Modules API - dynamic function calls? (same for themes?)
6. ???

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] admin menu

[Wolfgang <wol...@ex...>]

> probably we should just restyle the whole admin interface with some  
> yui (yahoo ui) ajax stuff. Some pages are already coded with this.

I planned to extend my small yahoo ui xquery module (yahoo-ui.xql) to 
support menubars, so we could use one hierarchical menu bar on the top 
of the page.

> otherwise.... xforms/serverside could do the trick?

I also added xforms to the admin webapp today (see the "Install Tools" 
menu item). But I guess implementing the entire webapp in XForms would 
be a bit too much. Anyway, we can mix both, XForms for the more complex 
user input forms, yahoo ui for the general interface.

Wolfgang

Re: [Exist-development] admin menu

[Dannes W. <di...@ex...>]

On 31 Oct 2009, at 21:52 , Dmitriy Shabanov wrote:

> The menu @admin page getting more and more elements and they are  
> mixed.
> Should we split admin menu (...exist/admin...) to groups (order them):
> Home?, Administration, Maintenance, Setup, Logout with subelements?
>
> Delete "Select a Page"?

probably we should just restyle the whole admin interface with some  
yui (yahoo ui) ajax stuff. Some pages are already coded with this.
otherwise.... xforms/serverside could do the trick?


Kind regards

Dannes

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

Re: [Exist-development] eXist & jetty

[Dannes W. <di...@ex...>]

On 31 Oct 2009, at 21:19 , Dmitriy Shabanov wrote:

> On Sat, 2009-10-31 at 21:11 +0100, Dannes Wessels wrote:
>> Dmitriy
>>
>> cool..... that enables us to have the newest jetty (7???) in exist
>> 1.4.1 ; I read that performance is better.....
>
> the answer should be yes, servlet API same
>
> Do we need JSP?

IMO..... no. we have some jsp code, but it is not really usable. we  
have xquery, right?

Kind regards

Dannes

--
eXist-db Native XML Database - http://exist-db.org
Join us on linked-in: http://www.linkedin.com/groups?gid=35624

[Exist-development] admin menu

[Dmitriy S. <sha...@gm...>]

Hi,

The menu @admin page getting more and more elements and they are mixed.
Should we split admin menu (...exist/admin...) to groups (order them):
Home?, Administration, Maintenance, Setup, Logout with subelements?

Delete "Select a Page"?

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] eXist & jetty

[Dmitriy S. <sha...@gm...>]

On Sat, 2009-10-31 at 21:46 +0100, Dannes Wessels wrote:
> On 31 Oct 2009, at 21:19 , Dmitriy Shabanov wrote:
> 
> > On Sat, 2009-10-31 at 21:11 +0100, Dannes Wessels wrote:
> >> Dmitriy
> >>
> >> cool..... that enables us to have the newest jetty (7???) in exist
> >> 1.4.1 ; I read that performance is better.....
> >
> > the answer should be yes, servlet API same
> >
> > Do we need JSP?
> 
> IMO..... no. we have some jsp code, but it is not really usable. we  
> have xquery, right?

+1 There are not much code.

-- 
Cheers,

Dmitriy Shabanov

Re: [Exist-development] eXist & jetty

[Dmitriy S. <sha...@gm...>]

On Sun, 2009-11-01 at 01:21 +0500, Dmitriy Shabanov wrote:
> On Sun, 2009-11-01 at 01:19 +0500, Dmitriy Shabanov wrote:
> > On Sat, 2009-10-31 at 21:11 +0100, Dannes Wessels wrote:
> > > Dmitriy
> > > 
> > > cool..... that enables us to have the newest jetty (7???) in exist
> > > 1.4.1 ; I read that performance is better.....
> 
> Well, the problem I see is "Almost stable"
> http://docs.codehaus.org/display/JETTY/Downloading+Jetty

+ http://wiki.eclipse.org/Jetty/Starting/Porting_to_Jetty_7

that mean a lot of refactoring (not much any way ;)

> > the answer should be yes, servlet API same
> > 
> > Do we need JSP?
> > 
-- 
Cheers,

Dmitriy Shabanov