Sedna Native XML Database

Hello,

Main new features in Sedna  3.0 are:
1) a lot of critical bug fixes
2) improved transaction/recovery based on versioning mechanism - 100%
recoverable

So it will be sort of unbreakable Sedna!

Maxim
Sedna team

2008/3/25, Steve Howe <howesteve@...>:
>
> Hello,
>
> I've seen tips that Sedna 3.0 will be released soon. Could you give us a
> glimpse of what new features will be out ? :)
>
> Thanks !
>
> --
> Best Regards,
> Steve Howe
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Sedna-discussion mailing list
> Sedna-discussion@...
> https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>

Hello all,
> Hello,
>
> Main new features in Sedna  3.0 are:
> 1) a lot of critical bug fixes
> 2) improved transaction/recovery based on versioning mechanism - 100%
> recoverable
>
> So it will be sort of unbreakable Sedna!
Very nice !

Any news on any of these:

- Hot backups
- Binary blobs
- Automatic index usage
- More modest/automated memory usage

Thanks for the info.

-- 
Best Regards,
Steve Howe

Hi,

please find my comments below.

2008/3/27, Steve Howe <howesteve@...>:
>
> Hello all,
>
> > Hello,
> >
> > Main new features in Sedna  3.0 are:
> > 1) a lot of critical bug fixes
> > 2) improved transaction/recovery based on versioning mechanism - 100%
> > recoverable
> >
> > So it will be sort of unbreakable Sedna!
>
> Very nice !
>
> Any news on any of these:
>
> - Hot backups


In Sedna  3.0 you can run read-only transaction which will not conflict with
any write transactions. So it allows you to run export utility which will
not stop other transactions. It is sort of Hot Backup. We are also thinking
about implementing it at physical level (i.e. copying data blocks instead of
exporting to XML) to speed up recovery process. Physical-level recovery can
be implemented using the same mechanism so I hope we will do it soon.

   - Binary blobs
Binary Blobs is not implemented yet. I think it will be one of our next
features.

- Automatic index usage



No. We don't want to implement this. It requires complex optimization which
is error-prone. Also we believe that this feature is not critical.

- More modest/automated memory usage


What do you mean under this?

Maxim
Sedna team

Hello Maxim Grinev,

> In Sedna  3.0 you can run read-only transaction which will not conflict
> with any write transactions. So it allows you to run export utility which
> will not stop other transactions. It is sort of Hot Backup. We are also
> thinking about implementing it at physical level (i.e. copying data blocks
> instead of exporting to XML) to speed up recovery process. Physical-level
> recovery can be implemented using the same mechanism so I hope we will do
> it soon.
Currently I was afraid of deplying anything on real field since I'd have to 
stop the database (and application) in order to backup the data.

>    - Binary blobs
> Binary Blobs is not implemented yet. I think it will be one of our next
> features.
That's going to be great! Binary storage is the Achilles' heel of XML 
databases.

> - Automatic index usage
> No. We don't want to implement this. It requires complex optimization which
> is error-prone. Also we believe that this feature is not critical.
I understand, however it would be a nice feature, specially for newcomers.

> - More modest/automated memory usage
> What do you mean under this?
Well, Sedna uses a lot of ram memory - 100Mb for each database. Is that really 
necessary ? I think it would manage it's memory as needed, dinamically, and 
being able to shrink/grow it's own bufffers/cache as needed. If you take 
PostgreSQL, for instance, it has global memory configurations, no matter how 
many databases are being handled. With the current Sedna implementation, if I 
have 5 databases, I'd be using 500Mb of ram, which is a lot if you use a 
remote VPS/dedicated server. I believe 100Mb could be much larger then data 
managed by Sedna on many real world databases... so why fixing on that 
length ?

-- 
Best Regards,
Steve Howe

Hello Steve,

I believe we could fix the memory issue. This will require complex
redesign of Sedna. Basicly we have to change the responsibility and
communication of Sedna processes. Currently there exists a separate
process for each database running the separate instance of buffer
manager. This leads to 100MB (when default configuration is in effect)
RAM used per DB - no matter how high the actual load is.

We could rearrange things and merge all SM-s into a single process.
This is a lot of work. I am willing to make this changes - along with
other enhancements to the buffer manager and overal cleanup of Sedna
system design. However I need a solid reasoning to convince the team
to invest effort in this project. Any ideas?

Mejedi, Sedna Team

2008/3/30, Steve Howe <howesteve@...>:
> Hello Maxim Grinev,
>
>
>  > In Sedna  3.0 you can run read-only transaction which will not conflict
>  > with any write transactions. So it allows you to run export utility which
>  > will not stop other transactions. It is sort of Hot Backup. We are also
>  > thinking about implementing it at physical level (i.e. copying data blocks
>  > instead of exporting to XML) to speed up recovery process. Physical-level
>  > recovery can be implemented using the same mechanism so I hope we will do
>  > it soon.
>
> Currently I was afraid of deplying anything on real field since I'd have to
>  stop the database (and application) in order to backup the data.
>
>
>  >    - Binary blobs
>  > Binary Blobs is not implemented yet. I think it will be one of our next
>  > features.
>
> That's going to be great! Binary storage is the Achilles' heel of XML
>  databases.
>
>
>  > - Automatic index usage
>  > No. We don't want to implement this. It requires complex optimization which
>  > is error-prone. Also we believe that this feature is not critical.
>
> I understand, however it would be a nice feature, specially for newcomers.
>
>
>  > - More modest/automated memory usage
>  > What do you mean under this?
>
> Well, Sedna uses a lot of ram memory - 100Mb for each database. Is that really
>  necessary ? I think it would manage it's memory as needed, dinamically, and
>  being able to shrink/grow it's own bufffers/cache as needed. If you take
>  PostgreSQL, for instance, it has global memory configurations, no matter how
>  many databases are being handled. With the current Sedna implementation, if I
>  have 5 databases, I'd be using 500Mb of ram, which is a lot if you use a
>  remote VPS/dedicated server. I believe 100Mb could be much larger then data
>  managed by Sedna on many real world databases... so why fixing on that
>  length ?
>
>
>  --
>  Best Regards,
>  Steve Howe
>
>  -------------------------------------------------------------------------
>  Check out the new SourceForge.net Marketplace.
>  It's the best place to buy or sell services for
>  just about anything Open Source.
>  http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
>  _______________________________________________
>  Sedna-discussion mailing list
>  Sedna-discussion@...
>  https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>

Hello ZVN,
> Hello Steve,
>
> I believe we could fix the memory issue. This will require complex
> redesign of Sedna. Basicly we have to change the responsibility and
> communication of Sedna processes. Currently there exists a separate
> process for each database running the separate instance of buffer
> manager. This leads to 100MB (when default configuration is in effect)
> RAM used per DB - no matter how high the actual load is.
>
> We could rearrange things and merge all SM-s into a single process.
> This is a lot of work. I am willing to make this changes - along with
> other enhancements to the buffer manager and overal cleanup of Sedna
> system design. However I need a solid reasoning to convince the team
> to invest effort in this project. Any ideas?
I think I do: without that kind of management, we won't ever see Sedna 
deployed in the wild. If you have a shared server, for instance, you can 
safely deploy PostgreSQL or MySQL because they can manage hundreds of 
databases at once, using the memory you reserve for them. Sedna however will 
choke the server's memory with only a few databases. That means expensive 
(dedicated) hardware will have to be used for running Sedna.

I have developed a web framework in Python which uses Sedna and lxml [1] but I 
can't see it deployed on actual hosting providers because of the high Sedna 
memory requirements. Most VPS plans start with 64 or 128Mb of ram, for 
instance. 

Besides, it's a question of efficiency. Why wasting more memory then needed ? 

I'd however keep each database connection running in a separate process and 
use a send process that would be the inter-process shared buffers manager. 
This is the approach PostgreSQL uses, and will avoid a crashed connection to 
shut down all the other connections or the whole database system. A similar 
implementation could be done in a single process using a buffers manager and 
threaded connections, however that could be a bit harder to implement due to 
threads compatibility issues among different operating systems.

-- 
Best Regards,
Steve Howe

[1] http://codespeak.net/lxml/

Hi Steve,

>    - Binary blobs
> > Binary Blobs is not implemented yet. I think it will be one of our next
> > features.
> That's going to be great! Binary storage is the Achilles' heel of XML
> databases.
>

Currently you can use BLOBs impelemented through XML Schema binary types
using XML:DB API by Charles Foster (http://www.cfoster.net/sedna).


> Well, Sedna uses a lot of ram memory - 100Mb for each database. Is that
> really
> necessary ?


100Mb is default value. It can be adjusted through "*--bufs-num*" command
line parameter.


> I think it would manage it's memory as needed, dinamically, and
> being able to shrink/grow it's own bufffers/cache as needed. If you take
> PostgreSQL, for instance, it has global memory configurations, no matter
> how
> many databases are being handled. With the current Sedna implementation,
> if I
> have 5 databases, I'd be using 500Mb of ram, which is a lot if you use a
> remote VPS/dedicated server.


Do you really need 5 databases?


Ivan Shcheklein,
Sedna Team

Hi,

I have to agree with Steve about memory issues, I currently use 2 hosting
packages.

1 is a high cost dedicated root server, and can install anything I want.
2 is a shared machine with my own Tomcat server, I am given 2 MySQL
Databases each running within the same server.

I use Sedna on the dedicated root server, but am still not happy with its
memory consumption.

As far as a shared machine goes, installing and running Sedna is just not
a option at the moment whether doing it myself or the host provider
themselves running 1 server with multiple databases. In this environment I
am currently forced to use eXist instead.

Regards,

Charles

> Hello ZVN,
>> Hello Steve,
>>
>> I believe we could fix the memory issue. This will require complex
>> redesign of Sedna. Basicly we have to change the responsibility and
>> communication of Sedna processes. Currently there exists a separate
>> process for each database running the separate instance of buffer
>> manager. This leads to 100MB (when default configuration is in effect)
>> RAM used per DB - no matter how high the actual load is.
>>
>> We could rearrange things and merge all SM-s into a single process.
>> This is a lot of work. I am willing to make this changes - along with
>> other enhancements to the buffer manager and overal cleanup of Sedna
>> system design. However I need a solid reasoning to convince the team
>> to invest effort in this project. Any ideas?
> I think I do: without that kind of management, we won't ever see Sedna
> deployed in the wild. If you have a shared server, for instance, you can
> safely deploy PostgreSQL or MySQL because they can manage hundreds of
> databases at once, using the memory you reserve for them. Sedna however
> will
> choke the server's memory with only a few databases. That means expensive
> (dedicated) hardware will have to be used for running Sedna.
>
> I have developed a web framework in Python which uses Sedna and lxml [1]
> but I
> can't see it deployed on actual hosting providers because of the high
> Sedna
> memory requirements. Most VPS plans start with 64 or 128Mb of ram, for
> instance.
>
> Besides, it's a question of efficiency. Why wasting more memory then
> needed ?
>
> I'd however keep each database connection running in a separate process
> and
> use a send process that would be the inter-process shared buffers manager.
> This is the approach PostgreSQL uses, and will avoid a crashed connection
> to
> shut down all the other connections or the whole database system. A
> similar
> implementation could be done in a single process using a buffers manager
> and
> threaded connections, however that could be a bit harder to implement due
> to
> threads compatibility issues among different operating systems.
>
> --
> Best Regards,
> Steve Howe
>
> [1] http://codespeak.net/lxml/
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> Sedna-discussion mailing list
> Sedna-discussion@...
> https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>

Hello Ivan,

> Currently you can use BLOBs impelemented through XML Schema binary types
> using XML:DB API by Charles Foster (http://www.cfoster.net/sedna).
I don't use Perl - how is that implemented, using base64 ? The overhead would 
be prohibitive, specially on large streams. Try to implement, let's say, a 
web gallery using that. And it would make the databases very large as well.

> 100Mb is default value. It can be adjusted through "*--bufs-num*" command
> line parameter.
I know, I read the docs - but it would still be per database and not global.

> Do you really need 5 databases?
That depends. What if I am a hosting provider willing to offer Sedna hosting ? 
I'd need 500 databases or even more. That is why you don't see any company 
offering Sedna hosting. Besides - yes, I could easily need 5 or more 
databases for running own different applications. In fact, I already do use 
4.

-- 
Best Regards,
Steve Howe

Hello Ivan,

> Currently you can use BLOBs impelemented through XML Schema binary types
> using XML:DB API by Charles Foster (http://www.cfoster.net/sedna).
Ok, I just saw it's Java, and I don't use it as well, however the same 
question applies, how is it stored internally in Sedna: if it's base64-based 
or something like that, it's useless except as a quick (but slow) hack.

-- 
Best Regards,
Steve Howe

Hello Steve,

       I think that the management of memory (Sedna) is not efficient, but
it has improved considerably in recent versions. It's important to remember
that Sedna is a "recent" system. Moreover, among all Native XML Databases,
Sedna is the only one that supports all the features of a traditional
database.
      I believe that the Sedna Team will improve the management of memory
and other features soon.


On Mon, Mar 31, 2008 at 10:42 AM, Steve Howe <howesteve@...> wrote:

> Hello Ivan,
>
> > Currently you can use BLOBs impelemented through XML Schema binary types
> > using XML:DB API by Charles Foster (http://www.cfoster.net/sedna).
> Ok, I just saw it's Java, and I don't use it as well, however the same
> question applies, how is it stored internally in Sedna: if it's
> base64-based
> or something like that, it's useless except as a quick (but slow) hack.
>
> --
> Best Regards,
> Steve Howe
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> Sedna-discussion mailing list
> Sedna-discussion@...
> https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>



-- 
Flávio R. C. Sousa, Prof.
Federal University of Ceará, Brazil
http://www.great.ufc.br/~flavio

Hi Sedna Team,

Do you have an expected release date for 3.0? 

I am hoping to assign one of our engineers to evaluate using Sedna as a potential back-end of our application - however as we are OSX based we can not do this until we have an OSX version that we can evaluate.

I ask about a release date as we are coming to the end of our current development cycle and it is a really good time to start this kind of investigative/research work.

Best regards

Alan Shouls
_________________________________________________________________
Pack up or back up–use SkyDrive to transfer files or keep extra copies. Learn how.
hthttp://www.windowslive.com/skydrive/overview.html?ocid=TXT_TAGLM_WL_Refresh_skydrive_packup_042008

Charles and Steve,

having read this discussion I also have to agree with you that Sedna should
share buffers between databases.
We should think about it and include it in our to-do list.

But please note that:
1) Memory consumption of XML databases will alsways be higher than in
relational databases. In rdb everything is more or less predictable. If we
read/modify an item (which is a tuple in rdb) we alway deal with only one
page. If we read/modify an item in XML database it might occupy
unpredictable number of pages. So to be efficient a database should have
relatively large buffers (100 Mb  or even more). So we are talking about
sharing buffers, not reducing its size. Is it ok for your applications?
2) As ZNV mentioned it will be not easy to implement it in Sedna but we
should do it.

Maxim Grinev
Sedna team




2008/3/31, Charles Foster <charles@...>:
>
> Hi,
>
> I have to agree with Steve about memory issues, I currently use 2 hosting
> packages.
>
> 1 is a high cost dedicated root server, and can install anything I want.
> 2 is a shared machine with my own Tomcat server, I am given 2 MySQL
> Databases each running within the same server.
>
> I use Sedna on the dedicated root server, but am still not happy with its
> memory consumption.
>
> As far as a shared machine goes, installing and running Sedna is just not
> a option at the moment whether doing it myself or the host provider
> themselves running 1 server with multiple databases. In this environment I
> am currently forced to use eXist instead.
>
> Regards,
>
>
> Charles
>
>
> > Hello ZVN,
> >> Hello Steve,
> >>
> >> I believe we could fix the memory issue. This will require complex
> >> redesign of Sedna. Basicly we have to change the responsibility and
> >> communication of Sedna processes. Currently there exists a separate
> >> process for each database running the separate instance of buffer
> >> manager. This leads to 100MB (when default configuration is in effect)
> >> RAM used per DB - no matter how high the actual load is.
> >>
> >> We could rearrange things and merge all SM-s into a single process.
> >> This is a lot of work. I am willing to make this changes - along with
> >> other enhancements to the buffer manager and overal cleanup of Sedna
> >> system design. However I need a solid reasoning to convince the team
> >> to invest effort in this project. Any ideas?
> > I think I do: without that kind of management, we won't ever see Sedna
> > deployed in the wild. If you have a shared server, for instance, you can
> > safely deploy PostgreSQL or MySQL because they can manage hundreds of
> > databases at once, using the memory you reserve for them. Sedna however
> > will
> > choke the server's memory with only a few databases. That means
> expensive
> > (dedicated) hardware will have to be used for running Sedna.
> >
> > I have developed a web framework in Python which uses Sedna and lxml [1]
> > but I
> > can't see it deployed on actual hosting providers because of the high
> > Sedna
> > memory requirements. Most VPS plans start with 64 or 128Mb of ram, for
> > instance.
> >
> > Besides, it's a question of efficiency. Why wasting more memory then
> > needed ?
> >
> > I'd however keep each database connection running in a separate process
> > and
> > use a send process that would be the inter-process shared buffers
> manager.
> > This is the approach PostgreSQL uses, and will avoid a crashed
> connection
> > to
> > shut down all the other connections or the whole database system. A
> > similar
> > implementation could be done in a single process using a buffers manager
> > and
> > threaded connections, however that could be a bit harder to implement
> due
> > to
> > threads compatibility issues among different operating systems.
> >
> > --
> > Best Regards,
> > Steve Howe
> >
> > [1] http://codespeak.net/lxml/
> >
>
> >
> -------------------------------------------------------------------------
> > Check out the new SourceForge.net Marketplace.
> > It's the best place to buy or sell services for
> > just about anything Open Source.
> >
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> > _______________________________________________
> > Sedna-discussion mailing list
> > Sedna-discussion@...
> > https://lists.sourceforge.net/lists/listinfo/sedna-discussion
> >
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> Sedna-discussion mailing list
> Sedna-discussion@...
> https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>

Charles,

I have a question about "shared machine" as I don't have a lot of experience
with using hosting packages. If you rent a shared machine and suppose that
the Sedna memory consumption is reduced, may you install Sedna on it? Or it
can be done only by the host provider?

And can you see any other problems (except memory consumption) of running
Sedna on a shared machine?

Thanks,
Maxim Grinev
Sedna team

2008/3/31, Charles Foster <charles@...>:
>
> Hi,
>
> I have to agree with Steve about memory issues, I currently use 2 hosting
> packages.
>
> 1 is a high cost dedicated root server, and can install anything I want.
> 2 is a shared machine with my own Tomcat server, I am given 2 MySQL
> Databases each running within the same server.
>
> I use Sedna on the dedicated root server, but am still not happy with its
> memory consumption.
>
> As far as a shared machine goes, installing and running Sedna is just not
> a option at the moment whether doing it myself or the host provider
> themselves running 1 server with multiple databases. In this environment I
> am currently forced to use eXist instead.
>
> Regards,
>
>
> Charles
>
>
> > Hello ZVN,
> >> Hello Steve,
> >>
> >> I believe we could fix the memory issue. This will require complex
> >> redesign of Sedna. Basicly we have to change the responsibility and
> >> communication of Sedna processes. Currently there exists a separate
> >> process for each database running the separate instance of buffer
> >> manager. This leads to 100MB (when default configuration is in effect)
> >> RAM used per DB - no matter how high the actual load is.
> >>
> >> We could rearrange things and merge all SM-s into a single process.
> >> This is a lot of work. I am willing to make this changes - along with
> >> other enhancements to the buffer manager and overal cleanup of Sedna
> >> system design. However I need a solid reasoning to convince the team
> >> to invest effort in this project. Any ideas?
> > I think I do: without that kind of management, we won't ever see Sedna
> > deployed in the wild. If you have a shared server, for instance, you can
> > safely deploy PostgreSQL or MySQL because they can manage hundreds of
> > databases at once, using the memory you reserve for them. Sedna however
> > will
> > choke the server's memory with only a few databases. That means
> expensive
> > (dedicated) hardware will have to be used for running Sedna.
> >
> > I have developed a web framework in Python which uses Sedna and lxml [1]
> > but I
> > can't see it deployed on actual hosting providers because of the high
> > Sedna
> > memory requirements. Most VPS plans start with 64 or 128Mb of ram, for
> > instance.
> >
> > Besides, it's a question of efficiency. Why wasting more memory then
> > needed ?
> >
> > I'd however keep each database connection running in a separate process
> > and
> > use a send process that would be the inter-process shared buffers
> manager.
> > This is the approach PostgreSQL uses, and will avoid a crashed
> connection
> > to
> > shut down all the other connections or the whole database system. A
> > similar
> > implementation could be done in a single process using a buffers manager
> > and
> > threaded connections, however that could be a bit harder to implement
> due
> > to
> > threads compatibility issues among different operating systems.
> >
> > --
> > Best Regards,
> > Steve Howe
> >
> > [1] http://codespeak.net/lxml/
> >
>
> >
> -------------------------------------------------------------------------
> > Check out the new SourceForge.net Marketplace.
> > It's the best place to buy or sell services for
> > just about anything Open Source.
> >
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> > _______________________________________________
> > Sedna-discussion mailing list
> > Sedna-discussion@...
> > https://lists.sourceforge.net/lists/listinfo/sedna-discussion
> >
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> Sedna-discussion mailing list
> Sedna-discussion@...
> https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>

Understood, one thing I'd like to see in the future is support for
collections that only accept documents that conform to a user defined XML
Schema.

Not only would this be beneficial in terms of developers using Sedna
knowing whether or not an XML document is valid by the Database
complaining when they tried to insert or update.

But surely this would also help with things like Sedna knowing how to
index the data easily, this surely would reduce the unpredictability
problem too?

Regards,

Charles

> Charles and Steve,
>
> having read this discussion I also have to agree with you that Sedna
> should
> share buffers between databases.
> We should think about it and include it in our to-do list.
>
> But please note that:
> 1) Memory consumption of XML databases will alsways be higher than in
> relational databases. In rdb everything is more or less predictable. If we
> read/modify an item (which is a tuple in rdb) we alway deal with only one
> page. If we read/modify an item in XML database it might occupy
> unpredictable number of pages. So to be efficient a database should have
> relatively large buffers (100 Mb  or even more). So we are talking about
> sharing buffers, not reducing its size. Is it ok for your applications?
> 2) As ZNV mentioned it will be not easy to implement it in Sedna but we
> should do it.
>
> Maxim Grinev
> Sedna team
>
>
>
>
> 2008/3/31, Charles Foster <charles@...>:
>>
>> Hi,
>>
>> I have to agree with Steve about memory issues, I currently use 2
>> hosting
>> packages.
>>
>> 1 is a high cost dedicated root server, and can install anything I want.
>> 2 is a shared machine with my own Tomcat server, I am given 2 MySQL
>> Databases each running within the same server.
>>
>> I use Sedna on the dedicated root server, but am still not happy with
>> its
>> memory consumption.
>>
>> As far as a shared machine goes, installing and running Sedna is just
>> not
>> a option at the moment whether doing it myself or the host provider
>> themselves running 1 server with multiple databases. In this environment
>> I
>> am currently forced to use eXist instead.
>>
>> Regards,
>>
>>
>> Charles
>>
>>
>> > Hello ZVN,
>> >> Hello Steve,
>> >>
>> >> I believe we could fix the memory issue. This will require complex
>> >> redesign of Sedna. Basicly we have to change the responsibility and
>> >> communication of Sedna processes. Currently there exists a separate
>> >> process for each database running the separate instance of buffer
>> >> manager. This leads to 100MB (when default configuration is in
>> effect)
>> >> RAM used per DB - no matter how high the actual load is.
>> >>
>> >> We could rearrange things and merge all SM-s into a single process.
>> >> This is a lot of work. I am willing to make this changes - along with
>> >> other enhancements to the buffer manager and overal cleanup of Sedna
>> >> system design. However I need a solid reasoning to convince the team
>> >> to invest effort in this project. Any ideas?
>> > I think I do: without that kind of management, we won't ever see Sedna
>> > deployed in the wild. If you have a shared server, for instance, you
>> can
>> > safely deploy PostgreSQL or MySQL because they can manage hundreds of
>> > databases at once, using the memory you reserve for them. Sedna
>> however
>> > will
>> > choke the server's memory with only a few databases. That means
>> expensive
>> > (dedicated) hardware will have to be used for running Sedna.
>> >
>> > I have developed a web framework in Python which uses Sedna and lxml
>> [1]
>> > but I
>> > can't see it deployed on actual hosting providers because of the high
>> > Sedna
>> > memory requirements. Most VPS plans start with 64 or 128Mb of ram, for
>> > instance.
>> >
>> > Besides, it's a question of efficiency. Why wasting more memory then
>> > needed ?
>> >
>> > I'd however keep each database connection running in a separate
>> process
>> > and
>> > use a send process that would be the inter-process shared buffers
>> manager.
>> > This is the approach PostgreSQL uses, and will avoid a crashed
>> connection
>> > to
>> > shut down all the other connections or the whole database system. A
>> > similar
>> > implementation could be done in a single process using a buffers
>> manager
>> > and
>> > threaded connections, however that could be a bit harder to implement
>> due
>> > to
>> > threads compatibility issues among different operating systems.
>> >
>> > --
>> > Best Regards,
>> > Steve Howe
>> >
>> > [1] http://codespeak.net/lxml/
>> >
>>
>> >
>> -------------------------------------------------------------------------
>> > Check out the new SourceForge.net Marketplace.
>> > It's the best place to buy or sell services for
>> > just about anything Open Source.
>> >
>> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
>> > _______________________________________________
>> > Sedna-discussion mailing list
>> > Sedna-discussion@...
>> > https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>> >
>>
>>
>> -------------------------------------------------------------------------
>> Check out the new SourceForge.net Marketplace.
>> It's the best place to buy or sell services for
>> just about anything Open Source.
>>
>> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
>> _______________________________________________
>> Sedna-discussion mailing list
>> Sedna-discussion@...
>> https://lists.sourceforge.net/lists/listinfo/sedna-discussion
>>
>

Hello Maxim Grinev,
> Charles and Steve,
>
> having read this discussion I also have to agree with you that Sedna should
> share buffers between databases.
> We should think about it and include it in our to-do list.
>
> But please note that:
> 1) Memory consumption of XML databases will alsways be higher than in
> relational databases. In rdb everything is more or less predictable. If we
> read/modify an item (which is a tuple in rdb) we alway deal with only one
> page. If we read/modify an item in XML database it might occupy
> unpredictable number of pages. So to be efficient a database should have
> relatively large buffers (100 Mb  or even more). So we are talking about
> sharing buffers, not reducing its size. Is it ok for your applications?
> 2) As ZNV mentioned it will be not easy to implement it in Sedna but we
> should do it.
I understand and agree about the large buffers. Thanks for the effort but even 
if it's a large one, should be worthy and finally make it possible for Sedna 
to get the widespread adoption it deserves.

Thank you.

-- 
Best Regards,
Steve Howe

On Sun, Mar 30, 2008 at 5:08 PM, ZNV <mejedi@...> wrote:
>
>  We could rearrange things and merge all SM-s into a single process.
>  This is a lot of work. I am willing to make this changes - along with
>  other enhancements to the buffer manager and overal cleanup of Sedna
>  system design. However I need a solid reasoning to convince the team
>  to invest effort in this project. Any ideas?
>

As long as you're thinking of major architectural changes I recommend
you revisit process management with an eye to adopting a more orthodox
daemon design, as well as a hierarchical process mgmt structure. For
example, is there a real advantage to having the user explicitly
launch a separate program (essentially a peer process) for something
like creating a database?  IOW, if I understand the architecture, it's
a kind of peer-to-peer implementation of a hierarchical control
structure.  Is that design superior to a more standard design with
some kind of master module that accepts user requests and
creates/manages subprocesses or threads as needed?

Also I strongly recommend you make security a high priority.

-gregg