qdbm-users — QDBM's mailing list for all users

[qdbm-users] [PATCH] workaround for ruby1.9 binding

[Satoru K. <lu...@gm...>]

hi,

I try to build qdbm with ruby1.9, and attach a patch for ruby binding
simply fixing structs to pass compiling them.

regards,
-- 
KURASHIKI Satoru

[qdbm-users] zLib in windows

[Varun D. <va...@ma...>]

Hi,

Is there any way to build QDBM with zLib in windows? Also, is encryption 
of the db file possible? Any suggestions are welcome.

Thanks

-- 
Varun Dhussa
Product Architect
CE InfoSystems (P) Ltd
http://www.mapmyindia.com

Re: [qdbm-users] Build QDBM on Windows MSVC

[Varun D. <va...@ma...>]

Hi,

I was able to work it out. I needed to run vcvarsall.bat first. I am 
sorry to bother the group. I have no experience building stuff on Windows.

Thanks

Varun Dhussa
Product Architect
CE InfoSystems (P) Ltd
http://www.mapmyindia.com



Varun Dhussa wrote:
> Hi,
>
> I am new to QDBM. I need to build QDBM on Windows (32 bit) and use it 
> with a program in MSVC. Since this needs a static library, I will need 
> to compile it from source. However, I am not able to do this. Can you 
> please help me with the same.
>
> The documentation says that the paths need to be set. However, I could 
> not see any paths in the VCMakeFile. I set the VC++ path and tried to 
> compile it, but I get the following error message:
>
> F:\work\Searching\qdbm-1.8.77>nmake /f VCmakefile
>
> Microsoft (R) Program Maintenance Utility Version 9.00.30729.01
> Copyright (C) Microsoft Corporation.  All rights reserved.
>
> Build using dynamic threaded release configuration
>         cl /c /I "C:\Program Files\Microsoft Visual Studio 
> 9.0\VC\Include" /I "C
> :\Program Files\Microsoft Visual Studio 9.0\VC\PlatformSDK\Include" /I 
> "."  /nologo  /MD /W3 /O2 /FD /Fo".\tmp\\" /Fd".\tmp\\" /DNDEBUG 
> /D_CRT_SECURE_NO_DEPRECATE=1 depot.c
> NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio 
> 9.0\VC\bin\cl.EXE"' : return code '0xc0000135'
> Stop.
>
> Please help me with this.
>
> Regards
>
>   

[qdbm-users] Build QDBM on Windows MSVC

[Varun D. <va...@ma...>]

Hi,

I am new to QDBM. I need to build QDBM on Windows (32 bit) and use it 
with a program in MSVC. Since this needs a static library, I will need 
to compile it from source. However, I am not able to do this. Can you 
please help me with the same.

The documentation says that the paths need to be set. However, I could 
not see any paths in the VCMakeFile. I set the VC++ path and tried to 
compile it, but I get the following error message:

F:\work\Searching\qdbm-1.8.77>nmake /f VCmakefile

Microsoft (R) Program Maintenance Utility Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

Build using dynamic threaded release configuration
        cl /c /I "C:\Program Files\Microsoft Visual Studio 
9.0\VC\Include" /I "C
:\Program Files\Microsoft Visual Studio 9.0\VC\PlatformSDK\Include" /I 
"."  /nologo  /MD /W3 /O2 /FD /Fo".\tmp\\" /Fd".\tmp\\" /DNDEBUG 
/D_CRT_SECURE_NO_DEPRECATE=1 depot.c
NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio 
9.0\VC\bin\cl.EXE"' : return code '0xc0000135'
Stop.

Please help me with this.

Regards

-- 
Varun Dhussa
Product Architect
CE InfoSystems (P) Ltd
http://www.mapmyindia.com

Re: [qdbm-users] record seek time (!)

[Steve C. <sco...@gm...>]

Wow, this is embarrassing.

> Please discuss tokyocabinet here:
>
> http://sourceforge.net/forum/forum.php?forum_id=712453

Thanks for the pointer, I totally missed this.

> Some ideas:
> - Have you used 'optimize' ?

You mean this blindingly obvious method starting me right in the face
in the rdoc? ;-)  I ran an optimize with a large bucket size and it
seems to have helped a lot.

Thanks for your help and your patience, Sven.  I'll move my question
over to the TC forum.

-Steve

Re: [qdbm-users] record seek time (!)

[Sven H. <Sven.Hartrumpf@FernUni-Hagen.de>]

Fri, 27 Mar 2009 07:56:02 -0700, sconover wrote:
> I'm using qdbm's successor, tokyocabinet, and though it's a different
> codebase I'm hoping to find help here (there's no obvious place to ask
> this question in TC-world)

Please discuss tokyocabinet here:

http://sourceforge.net/forum/forum.php?forum_id=712453

Some ideas:
- Have you used 'optimize' ?
- Have you tried all the different cache options for tch in some
  systematic way?

Greetings
Sven

[qdbm-users] record seek time (!)

[Steve C. <sco...@gm...>]

Hi,

I'm using qdbm's successor, tokyocabinet, and though it's a different
codebase I'm hoping to find help here (there's no obvious place to ask
this question in TC-world).  And I'm guessing qdbm is subject to the
same issues.

I have a 1 million record hash database, keys are integers and values
are string ranging from 50-400k apiece for a total db file size of
3.5GB.  When I first "get" a random key the time to return is a
minimum of 30ms and up to 100ms.  This is really bad.  The second
"get" of the same key is near-instantaneous - < 1ms - so I'm assuming
that I'm seeing the effects of the disk cache.

But that first "get" is what needs to be fast.  Can anyone provide
insight as to what might be happening?  Does this sound normal?  Is it
in fact disk random-read performance that's the issue here?  Is it
just kind of an inviolable law of using a technology that's
disk-IO-based that you're subject to this kind of issue?  Or perhaps
I'm configuring my db incorrectly?

Any help very much appreciated.

Regards,
Steve

Re: [qdbm-users] Please help with elements count.

[Oleg P. <syb...@gm...>]

2009/3/11 Chris Dean <ct...@so...>

>
> > Thanks for help. Unfortunately i have about 20-30 requests for place per
> > second.
>
> It seems like there should be a clever solution to this problem, but I
> can't think of it.
>
> In the absence of something clever - If reads dominate, then the simple
> answer is to keep an ordered list in ram and do a binary search to find
> the score position.  Inserting new scores is more expensive of course.
> I suppose you could also use a flat file for the same purpose.


It's a gaming portal. Users play a lot. So the number of updates is much
more than selects.
(200 updates/s - 200 game matches per second are overed -  + 25 selects/s,
so any type of caching is useless). The portal is heavy loaded (60.000.000
pageloads/day) so usually
standart solutions do not work in my case.


>
>
> > Theoretically QDBM could know the number of keys under each leaf. For
> > example
>
> Right.  I don't think TC keeps that information around.  You could
> always code up a b+tree soultion with this feature (that was my first
> thought, but seems like to much work).


Yeah, especially for balancing b+tree.


>
>
> Cheers,
> Chris Dean
>
>

Re: [qdbm-users] Please help with elements count.

[Chris D. <ct...@so...>]

> Thanks for help. Unfortunately i have about 20-30 requests for place per
> second. 

It seems like there should be a clever solution to this problem, but I
can't think of it.

In the absence of something clever - If reads dominate, then the simple
answer is to keep an ordered list in ram and do a binary search to find
the score position.  Inserting new scores is more expensive of course.
I suppose you could also use a flat file for the same purpose.

> Theoretically QDBM could know the number of keys under each leaf. For
> example

Right.  I don't think TC keeps that information around.  You could
always code up a b+tree soultion with this feature (that was my first
thought, but seems like to much work).

Cheers,
Chris Dean

Re: [qdbm-users] Please help with elements count.

[Oleg P. <syb...@gm...>]

Thanks for help. Unfortunately i have about 20-30 requests for place per
second. Currently i'm using mysql heap table for this task (table with 2
columns - user and rating and request for place SELECT count(*) FROM table
WHERE rating > [myrating]) and it tooks about 0.5s to calculate the worst
case, but it heavy loads 2xQuad core Xeon :( So i'm looking for better
solution.

I tried berkeley DB with it's ->db_key_range function (tooks 0.00001 sec in
worst case) but it's too inaccurate for my task.

Theoretically QDBM could know the number of keys under each leaf. For
example

                        G
                    7 /   \  1
                    D      K
                3 /  3 \     \ 1
                B        F    M
             1/  1\   1/ 1\
            A   C    E   G

It could be easily maintained - just when new record is inserted,  make ++
to each leaf on the way to node's position in the tree.

In this case the place is simply the sum of all counts on the way from node
to root node.
But it seems that QDBM at least has no API for doing that.

2009/3/11 Chris Dean <ct...@so...>

>
> Oleg Pronin <syb...@gm...> writes:
> > Greetings. I'm new to QDBM and i'm stuck with a simple task:
> > I have a lot of numbers (several millions) and i need to calculate what
> > place does a number take in overall list according to my sort order.
>
> An interesting problem.  The only way that I can think of inside TC is
> to create a reverse index sorted by score and find where your score is
> in that set.  One way to do this is to use the Table Database and then
> query for your column.
>
> I tried it on my box with 10 million records and it took 9 seconds to
> find the middle value.
>
> Maybe someone else can think of a better solution.
>
> Code below.
>
> Cheers,
> Chris Dean
>
>
>
> #!/usr/bin/env perl
> #
> # 10 March 2009
>
> use strict;
> use warnings;
> use TokyoCabinet;
>
> sub populate {
>    my( $db, $how_many ) = @_;
>
>    for( my $i = 0; $i < $how_many; ++$i ) {
>        $db->put( $i, { score => $i } );
>    }
> }
>
> sub score_position_iter {
>    my( $db, $target_score ) = @_;
>
>    my $query = TokyoCabinet::TDBQRY->new( $db );
>    $query->setorder( "score", $query->QONUMASC );
>
>    my $i = 1;
>    my $found = 0;
>    $query->proc( sub {
>        my( $key, $cols ) = @_;
>
>        if( $cols->{score} == $target_score ) {
>            $found = $i;
>            return( $query->QPSTOP );
>        }
>
>        ++$i;
>
>        return( 0 );
>    } );
>
>    return( $found );
> }
>
> sub score_position_search {
>    my( $db, $target_score ) = @_;
>
>    my $query = TokyoCabinet::TDBQRY->new( $db );
>    $query->addcond( "score", $query->QCNUMLE, $target_score );
>
>    my $res = $query->search();
>    return( scalar( @{ $res } ) );
> }
>
> sub main {
>    # my $size = 10_000_000;
>    my $size = 1_000_000;
>
>    print "open db ...\n";
>    my $start = time();
>    my $db = TokyoCabinet::TDB->new();
>    my $existed = ( -f "scores.db" );
>    $db->open( "scores.db", $db->OWRITER | $db->OCREAT ) or die;
>    unless( $existed ) {
>        $db->setindex( "score", $db->ITDECIMAL );
>    }
>
>    my $n = time() - $start;
>    print "open db ... $n secs\n";
>
>    unless( $existed ) {
>        $start = time();
>        print "populating $size ...\n";
>        populate( $db, $size );
>
>        $n = time() - $start;
>        print "populate $size ... $n secs\n";
>    }
>
>    $start = time();
>    print "finding ...\n";
>    my $pos = score_position_search( $db, $size / 2 );
>    $n = time() - $start;
>    print "finding ... $n secs\n";
>
>    print "You have the ${pos}th best score out of $size scores\n";
>
>    $db->close() or die;
> }
>
> main();
>

Re: [qdbm-users] Please help with elements count.

[Chris D. <ct...@so...>]

Oleg Pronin <syb...@gm...> writes:
> Greetings. I'm new to QDBM and i'm stuck with a simple task:
> I have a lot of numbers (several millions) and i need to calculate what
> place does a number take in overall list according to my sort order.

An interesting problem.  The only way that I can think of inside TC is
to create a reverse index sorted by score and find where your score is
in that set.  One way to do this is to use the Table Database and then
query for your column.

I tried it on my box with 10 million records and it took 9 seconds to
find the middle value.

Maybe someone else can think of a better solution.

Code below. 

Cheers,
Chris Dean



#!/usr/bin/env perl
#
# 10 March 2009

use strict;
use warnings;
use TokyoCabinet;

sub populate {
    my( $db, $how_many ) = @_;

    for( my $i = 0; $i < $how_many; ++$i ) {
        $db->put( $i, { score => $i } );
    }
}

sub score_position_iter {
    my( $db, $target_score ) = @_;

    my $query = TokyoCabinet::TDBQRY->new( $db );
    $query->setorder( "score", $query->QONUMASC );

    my $i = 1;
    my $found = 0;
    $query->proc( sub {
        my( $key, $cols ) = @_;

        if( $cols->{score} == $target_score ) {
            $found = $i;
            return( $query->QPSTOP );
        }

        ++$i;

        return( 0 );
    } );

    return( $found );
}
        
sub score_position_search {
    my( $db, $target_score ) = @_;

    my $query = TokyoCabinet::TDBQRY->new( $db );
    $query->addcond( "score", $query->QCNUMLE, $target_score );

    my $res = $query->search();
    return( scalar( @{ $res } ) );
}
        
sub main {
    # my $size = 10_000_000;
    my $size = 1_000_000;

    print "open db ...\n";
    my $start = time();
    my $db = TokyoCabinet::TDB->new();
    my $existed = ( -f "scores.db" );
    $db->open( "scores.db", $db->OWRITER | $db->OCREAT ) or die;
    unless( $existed ) {
        $db->setindex( "score", $db->ITDECIMAL );
    }

    my $n = time() - $start;
    print "open db ... $n secs\n";

    unless( $existed ) {
        $start = time();
        print "populating $size ...\n";
        populate( $db, $size );

        $n = time() - $start;
        print "populate $size ... $n secs\n";
    }

    $start = time();
    print "finding ...\n";
    my $pos = score_position_search( $db, $size / 2 );
    $n = time() - $start;
    print "finding ... $n secs\n";

    print "You have the ${pos}th best score out of $size scores\n";

    $db->close() or die;
}

main();

[qdbm-users] Please help with elements count.

[Oleg P. <syb...@gm...>]

Greetings. I'm new to QDBM and i'm stuck with a simple task:
I have a lot of numbers (several millions) and i need to calculate what
place does a number take in overall list according to my sort order.
The numbers are ratings of users in a game. And i want to show user's place.

I.e. i need to do something like that for example:

for (my $i = 0; $i < 10000000; $i+=2) $db{$i} = $i;

print $db->some_cool_method(200000); #prints 100000

Is it possible without interating over all of the tree?

Re: [qdbm-users] random access

[Sven H. <Sven.Hartrumpf@FernUni-Hagen.de>]

Hi again.

Thu, 27 Nov 2008 09:33:51 -0500, derek wrote:

> I know that I can jump to a particular key, but what I need to do is
> essentially jump to a particular offset.

Sorry, I misunderstood your problem and now I don't see a simple solution.

> BTW, is Tokyocabinet backwards compatable with QDBM?

No, but conversion scripts are easy and safe, here
without any parameter tuning:

#!
# villa to tokyocabinet btree
# vl2tcb infile outfile

vltsv export $1 | tcbmgr importtsv $2



> I have over a million QDBM Villa files.

Impressive. Can you tell us a little bit about the sizes and numbers
of keys and values?

Sven

Re: [qdbm-users] random access

[Derek S. <de...@ho...>]

Hi Sven,

I know that I can jump to a particular key, but what I need to do is
essentially jump to a particular offset.

For example, say my B+ tree contains the following sequence of entries:

  apple banana carrot dog eat fruit gorilla hat igloo jump karate llama

I want to be able to display entries 6 to 8 without having to start at
"apple", and without knowing that #6 is "fruit".

I cannot make "fruit" have a key of "6", as if I add the word "fable" to
the database, then "fruit" will then become the 7th entry in the
sequence.

Thanks :)

-Derek

BTW, is Tokyocabinet backwards compatable with QDBM?  I have over a
million QDBM Villa files.


On Thu, 2008-11-27 at 09:37 +0100, Sven Hartrumpf wrote:
> Hi Derek.
> 
> Wed, 26 Nov 2008 12:06:45 -0500, derek wrote:
> > I was wondering if there is any feasible way to provide random access to
> > data within a B+ Tree Villa database?
> 
> Yes.
> 
> > Now, say I need to be able to quickly retrieve the middle 100 words
> > (words 50,000 to 50,099).  Is there a way to do this quickly, without
> > starting at the beginning of the list, and counting along until I get to
> > 50,000?
> 
> Please have a look at the functions named vlcur...
> For example, vlcurjump plus an iteration of vlcurval and vlcurnext should
> be a possibility.
> 
> BTW. Have you considered switching to tokyocabinet?
> 
> Sven

Re: [qdbm-users] random access

[Sven H. <Sven.Hartrumpf@FernUni-Hagen.de>]

Hi Derek.

Wed, 26 Nov 2008 12:06:45 -0500, derek wrote:
> I was wondering if there is any feasible way to provide random access to
> data within a B+ Tree Villa database?

Yes.

> Now, say I need to be able to quickly retrieve the middle 100 words
> (words 50,000 to 50,099).  Is there a way to do this quickly, without
> starting at the beginning of the list, and counting along until I get to
> 50,000?

Please have a look at the functions named vlcur...
For example, vlcurjump plus an iteration of vlcurval and vlcurnext should
be a possibility.

BTW. Have you considered switching to tokyocabinet?

Sven

[qdbm-users] random access

[Derek S. <de...@ho...>]

Greetings,

I was wondering if there is any feasible way to provide random access to
data within a B+ Tree Villa database?

For example, say I have 100,000 entries in a qdbm Villa database, sorted
alphabetically (say dictionary words for example), and I'm using this
type of database (Villa) because I want to be able to add and/or remove
words, and then to be able to retrieve them in sorted order.

Now, say I need to be able to quickly retrieve the middle 100 words
(words 50,000 to 50,099).  Is there a way to do this quickly, without
starting at the beginning of the list, and counting along until I get to
50,000?

Thanks :)

-Derek

Re: [qdbm-users] dpopen

[George N. <ge...@ne...>]

Mikio,

On Sun, Oct 5, 2008 at 4:16 AM, Mikio Hirabayashi
<mi...@us...> wrote:
> Hi,
>
>> It seems to me these checks should -not- be qualified with  !(omode &
>> DP_ONOLCK)
>>
>> Is this a bug or intended?
>
> It is intended behaviour.
> When DP_ONOLCK is specified, validation to the database format is not
> assured.  So, check is omitted.

Sometimes an invalid depot file will be passed to dpopen(); and it
doesn't fail.  I will "mode & ~DP_ONOLCK" in my wrapper function to
force the header check for now.

Is it possible to have the dpopen() documentation mention DP_ONOLCK
turns off depot header and file size checks, it currently only
mentions that it opens a database without locking.

TIA,
George.

Re: [qdbm-users] dpopen

[Mikio H. <mi...@us...>]

Hi,

> It seems to me these checks should -not- be qualified with  !(omode &
> DP_ONOLCK)
> 
> Is this a bug or intended?

It is intended behaviour.
When DP_ONOLCK is specified, validation to the database format is not
assured.  So, check is omitted.

Regards.

// Mikio Hirabayashi

[qdbm-users] dpopen

[George N. <ge...@ne...>]

Hi All,

I am a new user of qdbm,  I think there may be a bug in depot.c:255,
dpopen()   (from v.1.8.77)

  if(!(omode & DP_ONOLCK) &&
     ((dpbigendian() ? memcmp(hbuf, DP_MAGICNUMB, strlen(DP_MAGICNUMB)) != 0 :
       memcmp(hbuf, DP_MAGICNUML, strlen(DP_MAGICNUML)) != 0) ||
      *((int *)(hbuf + DP_FSIZOFF)) != fsiz)){
    close(fd);
    dpecodeset(DP_EBROKEN, __FILE__, __LINE__);
    return NULL;
  }

It seems to me these checks should -not- be qualified with  !(omode &
DP_ONOLCK)

Is this a bug or intended?

Thanks,
George.

[qdbm-users] how to anable zlib compression option in VC++?

[Allen C. <cp...@gm...>]

Hi,all,

Have any of you compiled the QDBM on Windows with a compression library?
Or is there any way to enable the option in VCmakefile?
Any suggestion or advise is welcome!

thanks,
Allen

Re: [qdbm-users] data migration from GDBM to QDBM

[Alejandro M. <am...@bq...>]

Hi,

a short answer from a mere user.

> For one of our applications we consider to switch from GDBM to QDBM.

please consider tokyocabinet for new development, as it is the natural
successor of qdbm.
http://tokyocabinet.sourceforge.net/


> The spec from QDBM mentions that:
> Hovel can not handle database files made by the original GDBM.
>
> Regarding this incompatibity I have 2 questions:
>
> does this require the explicit removal of the original GDBM files as part of
> the transistion?
> suppose we require data migration, is it easy to migrate GDBM data to QDBM?

the easiest imho is a massive export/import using CLI tools. but
anyway it should be easy.


Regards,
Alejandro Mery

[qdbm-users] data migration from GDBM to QDBM

[Ton W. <ton...@as...>]

Hello qdbm-users,
 
For one of our applications we consider to switch from GDBM to QDBM.
The spec from QDBM mentions that: 
Hovel can not handle database files made by the original GDBM.
 
Regarding this incompatibity I have 2 questions:

1.	does this require the explicit removal of the original GDBM
files as part of the transistion?
2.	suppose we require data migration, is it easy to migrate GDBM
data to QDBM?

regards,
 
Ton Willems
ASML Veldhoven


-- 
The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited.  Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities.  To the extent you are relying on this information, you are doing so at your own risk.   If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. ASML is neither liable for the proper and complete transmission of the information contained in this communication, nor for any delay in its receipt. 

Re: [qdbm-users] Locking

[Sven.Hartrumpf@FernUni-Hagen.de]

Mon, 26 Nov 2007 17:41:40 +0100 (CET), Sven wrote:

> Mon, 06 Aug 2007 11:06:13 +0200, ninive wrote:
> 
> > If I have an open QDBM-handle and the application cashes,
> > the lock remains and blocks all future requests.
> >
> > Is there a chance to set a "timeout" for locks?
> 
> I am also interested in such a feature.
> 
> And if not:
> How can I remove a (write) lock which remains when
> the application crashes?

I am still facing the problem that write locks remain.
Looking at other example programs, I would expect that programs like
dptsv should remove the lock (set in the dplock-call from dpopen)
at the end (for example, in dpclose?).

Greetings
Sven

[qdbm-users] Multi-threaded usage from C

[Prashanth B. <pra...@ma...>]

Hello,

I'm using QDBM (C version). I would like to know if it is safe to use 
multiple handles to the same DB from multiple threads in the same process.

On the QDBM C++ page, there is mention of a bug:
"QDBM has restrictions that two or more handles of the same database 
file should not be used by a process at the same time. So, when a 
database is used by two or more threads, open the database in the main 
thread and pass the handle to each thread."

However, earlier on the same page, it is mentioned that:
"While APIs for C are thread-safe unless plural threads do not share a 
database handle, APIs for C++ are thread-safe even if plural threads 
share a handle. This API is implemented depending on the POSIX thread 
package. If you use another thread package, you should write your own 
C++ wrapper. "

I'm confused - could some one please clarify?

Thanks,
Prashanth

Re: [qdbm-users] Locking

[Sven.Hartrumpf@FernUni-Hagen.de]

Mon, 06 Aug 2007 11:06:13 +0200, ninive wrote:

> If I have an open QDBM-handle and the application cashes,
> the lock remains and blocks all future requests.
>
> Is there a chance to set a "timeout" for locks?

I am also interested in such a feature.

And if not:
How can I remove a (write) lock which remains when
the application crashes?

Sven