naviserver-devel — Developer discussion

Re: [naviserver-devel] make memcheck / valgrind

[Stephen D. <sd...@gm...>]

On 11/21/08, Bernd Eidenschink <eid...@we...> wrote:
>
>  > No no. Errors are bad, there should be none.
>  >
>  > Stop teasing and show us them...   :-)
>
>
> Here you go:
>  http://www.kinetiqa.de/naviserver/memcheck.log.txt
>
>  TCL: 8.4.19 sources


This is weird, 8.4 also has errors? (you were using 8.5.5 before?)

8.4.19 works for me.

Are you compiling 32bit on a 64bit linux box?


Anyway, these kinds of things are usually errors (the message is from valgrind):

==18662== Invalid read of size 4
==18662==    at 0x40151E3: (within /lib/ld-2.7.so)
==18662==    by 0x4005C59: (within /lib/ld-2.7.so)
==18662==    by 0x4007A87: (within /lib/ld-2.7.so)
==18662==    by 0x4011533: (within /lib/ld-2.7.so)
==18662==    by 0x400D5C5: (within /lib/ld-2.7.so)
==18662==    by 0x4010F4D: (within /lib/ld-2.7.so)
==18662==    by 0x41E9C18: (within /lib/tls/i686/cmov/libdl-2.7.so)
==18662==    by 0x400D5C5: (within /lib/ld-2.7.so)
==18662==    by 0x41EA2BB: (within /lib/tls/i686/cmov/libdl-2.7.so)
==18662==    by 0x41E9B50: dlopen (in /lib/tls/i686/cmov/libdl-2.7.so)
==18662==    by 0x41CF245: TclpDlopen (tclLoadDl.c:78)
==18662==    by 0x419195C: Tcl_FSLoadFile (tclIOUtil.c:2791)
==18662==  Address 0x4fd2fc0 is 32 bytes inside a block of size 35 alloc'd
==18662==    at 0x4022AB8: malloc (vg_replace_malloc.c:207)
==18662==    by 0x4006FC4: (within /lib/ld-2.7.so)
==18662==    by 0x40079C9: (within /lib/ld-2.7.so)
==18662==    by 0x4011533: (within /lib/ld-2.7.so)
==18662==    by 0x400D5C5: (within /lib/ld-2.7.so)
==18662==    by 0x4010F4D: (within /lib/ld-2.7.so)
==18662==    by 0x41E9C18: (within /lib/tls/i686/cmov/libdl-2.7.so)
==18662==    by 0x400D5C5: (within /lib/ld-2.7.so)
==18662==    by 0x41EA2BB: (within /lib/tls/i686/cmov/libdl-2.7.so)
==18662==    by 0x41E9B50: dlopen (in /lib/tls/i686/cmov/libdl-2.7.so)
==18662==    by 0x41CF245: TclpDlopen (tclLoadDl.c:78)
==18662==    by 0x419195C: Tcl_FSLoadFile (tclIOUtil.c:2791)



>  BTW: "make install" fails because of "install-docs" in Makefile,
>  maybe it would make sense to change the Makefile to be more
>  aware of missing "dtplite"-missing situations.


The idea is that if you are building from a released tarball then the
built documentation is included and you don't need dtplite. If you're
building direct from the repo, you need dtplite (and autoconf, etc.).


>  Nice: The ns_thread.test(s) double the memory usage, saturate my box with
>  100% CPU load... but, once done, all falls back to where it started.


Well at least something's working!

Re: [naviserver-devel] nsdbpg

[Stephen D. <sd...@gm...>]

On 11/21/08, Bernd Eidenschink <eid...@we...> wrote:
>
>  I'm cursed. I never can compile a postgres driver for naviserver
>  from scratch.
>
>  Here's my tale - maybe you can hint me to an obvious mistake:
>
>  I compiled a Postgres 8.2.11 from source into
>  /opt/pgsql8.2.11
>
>  Then I checked out /modules/nsdbpg.
>
>  I changed in the Makefile the line
>  MODLIBS    = -lnsdb -lpq


You shouldn't need to edit the Makefile.

NAVISERVER points to the base of your naviserver install, and
POSTGRES points the base of your postgres install.

So,

$ make NAVISERVER=/usr/local/ns POSTGRES=/opt/pgsql8.2.11


(The Makefile uses -rpath to embed the location of postgres into
nsdbpg, so you shouldn't need to use LD_LIBRARY_PATH)

Re: [naviserver-devel] nsdbpg

[Vlad S. <vl...@cr...>]

change

MODLIBS = -L/usr/local/ns/include/nsdb.h -L/opt/pgsql8.2.11/lib/libpq.a

to

MODLIBS = -lnsdb -L/opt/pgsql8.2.11/lib -lpq


Bernd Eidenschink wrote:
> I'm cursed. I never can compile a postgres driver for naviserver
> from scratch.
> 
> Here's my tale - maybe you can hint me to an obvious mistake:
> 
> I compiled a Postgres 8.2.11 from source into
> /opt/pgsql8.2.11
> 
> Then I checked out /modules/nsdbpg.
> 
> I changed in the Makefile the line
> MODLIBS    = -lnsdb -lpq
> 
> to
> MODLIBS = -L/usr/local/ns/include/nsdb.h -L/opt/pgsql8.2.11/lib/libpq.a
> 
> and started make with the line
> make NAVISERVER=/usr/local/ns 
> CFLAGS="-I/opt/pgsql8.2.11/lib -I/opt/pgsql8.2.11/include -I/usr/local/ns/include"
> 
> It compiles. The only output is:
> gcc -I/opt/pgsql8.2.11/lib -I/opt/pgsql8.2.11/include -I/usr/local/ns/include   -c -o 
> nsdbpg.o nsdbpg.c
> gcc -I/opt/pgsql8.2.11/lib -I/opt/pgsql8.2.11/include -I/usr/local/ns/include   -c -o 
> tclcmds.o tclcmds.c
> tclcmds.c: In function ‘stream_actually_write’:
> tclcmds.c:830: warning: ‘Ns_ConnWrite’ is deprecated (declared 
> at /usr/local/ns/include/ns.h:1111)
> /bin/rm -Rf nsdbpg.so
> gcc -shared -I/opt/pgsql8.2.11/lib -I/opt/pgsql8.2.11/include -I/usr/local/ns/include  -L/usr/local/ns/lib -o 
> nsdbpg.so nsdbpg.o 
> tclcmds.o  -L/usr/local/ns/include/nsdb.h -L/opt/pgsql8.2.11/lib/libpq.a -lnsthread -lnsd -L/usr/local/ns/lib -ltcl8.4 -ldl -lgcc_s  -lieee -lm  -Wl,--export-dynamic  -L/usr/local/ns/lib -Wl,-rpath,:/usr/local/ns/lib
> 
> So this warning about Ns_ConnWrite - ignored.
> 
> No doing a "ldd nsdbpg.so"
> returns
> 
> linux-gate.so.1 =>  (0xb7fce000)
> libnsthread.so => /usr/local/ns/lib/libnsthread.so (0xb7fbe000)
> libnsd.so => /usr/local/ns/lib/libnsd.so (0xb7f46000)
> libtcl8.4.so => /usr/local/ns/lib/libtcl8.4.so (0xb7e90000)
> libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e87000)
> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7e7c000)
> libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7e57000)
> libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d08000)
> libz.so.1 => /usr/lib/libz.so.1 (0xb7cf2000)
> libcrypt.so.1 => /lib/tls/i686/cmov/libcrypt.so.1 (0xb7cc0000)
> libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7ca8000)
> /lib/ld-linux.so.2 (0xb7fcf000)
> 
> 
> Hm. On another server "libpq" and "libnsdb" come up,
> missing here.
> 
> When trying to run naviserver, called from my startscript
> with an exported LD_LIBRARY_PATH of
> 
> export 
> LD_LIBRARY_PATH=/usr/local/ns/lib:/usr/local/ns/lib/xotcl1.6.2:/usr/local/ns/bin:/opt/pgsql8.2.11/lib:/usr/local/lib
> 
> including the postgres directory, I get:
> 
> [21/Nov/2008:18:23:52][5641.b7c426b0][-main-] Error: 
> modload: /usr/local/ns/bin/nsdbpg.so: couldn't load 
> file "/usr/local/ns/bin/nsdbpg.so": /usr/local/ns/bin/nsdbpg.so: undefined 
> symbol: PQsetdbLogin
> 
> Can you help me, Veronica Mars?
> 
> Bernd.
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel

[naviserver-devel] nsdbpg

[Bernd E. <eid...@we...>]

I'm cursed. I never can compile a postgres driver for naviserver
from scratch.

Here's my tale - maybe you can hint me to an obvious mistake:

I compiled a Postgres 8.2.11 from source into
/opt/pgsql8.2.11

Then I checked out /modules/nsdbpg.

I changed in the Makefile the line
MODLIBS    = -lnsdb -lpq

to
MODLIBS = -L/usr/local/ns/include/nsdb.h -L/opt/pgsql8.2.11/lib/libpq.a

and started make with the line
make NAVISERVER=/usr/local/ns 
CFLAGS="-I/opt/pgsql8.2.11/lib -I/opt/pgsql8.2.11/include -I/usr/local/ns/include"

It compiles. The only output is:
gcc -I/opt/pgsql8.2.11/lib -I/opt/pgsql8.2.11/include -I/usr/local/ns/include   -c -o 
nsdbpg.o nsdbpg.c
gcc -I/opt/pgsql8.2.11/lib -I/opt/pgsql8.2.11/include -I/usr/local/ns/include   -c -o 
tclcmds.o tclcmds.c
tclcmds.c: In function ‘stream_actually_write’:
tclcmds.c:830: warning: ‘Ns_ConnWrite’ is deprecated (declared 
at /usr/local/ns/include/ns.h:1111)
/bin/rm -Rf nsdbpg.so
gcc -shared -I/opt/pgsql8.2.11/lib -I/opt/pgsql8.2.11/include -I/usr/local/ns/include  -L/usr/local/ns/lib -o 
nsdbpg.so nsdbpg.o 
tclcmds.o  -L/usr/local/ns/include/nsdb.h -L/opt/pgsql8.2.11/lib/libpq.a -lnsthread -lnsd -L/usr/local/ns/lib -ltcl8.4 -ldl -lgcc_s  -lieee -lm  -Wl,--export-dynamic  -L/usr/local/ns/lib -Wl,-rpath,:/usr/local/ns/lib

So this warning about Ns_ConnWrite - ignored.

No doing a "ldd nsdbpg.so"
returns

linux-gate.so.1 =>  (0xb7fce000)
libnsthread.so => /usr/local/ns/lib/libnsthread.so (0xb7fbe000)
libnsd.so => /usr/local/ns/lib/libnsd.so (0xb7f46000)
libtcl8.4.so => /usr/local/ns/lib/libtcl8.4.so (0xb7e90000)
libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e87000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7e7c000)
libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7e57000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d08000)
libz.so.1 => /usr/lib/libz.so.1 (0xb7cf2000)
libcrypt.so.1 => /lib/tls/i686/cmov/libcrypt.so.1 (0xb7cc0000)
libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7ca8000)
/lib/ld-linux.so.2 (0xb7fcf000)


Hm. On another server "libpq" and "libnsdb" come up,
missing here.

When trying to run naviserver, called from my startscript
with an exported LD_LIBRARY_PATH of

export 
LD_LIBRARY_PATH=/usr/local/ns/lib:/usr/local/ns/lib/xotcl1.6.2:/usr/local/ns/bin:/opt/pgsql8.2.11/lib:/usr/local/lib

including the postgres directory, I get:

[21/Nov/2008:18:23:52][5641.b7c426b0][-main-] Error: 
modload: /usr/local/ns/bin/nsdbpg.so: couldn't load 
file "/usr/local/ns/bin/nsdbpg.so": /usr/local/ns/bin/nsdbpg.so: undefined 
symbol: PQsetdbLogin

Can you help me, Veronica Mars?

Bernd.

Re: [naviserver-devel] make memcheck / valgrind

[Bernd E. <eid...@we...>]

> No no. Errors are bad, there should be none.
>
> Stop teasing and show us them...   :-)

Here you go:
http://www.kinetiqa.de/naviserver/memcheck.log.txt

TCL: 8.4.19 sources
Naviserver: trunk (CVS)

/tmp/tease/naviserver$ ./autogen.sh --enable-symbols --enable-threads --prefix=/tmp/ns --with-tcl=/tmp/ns/lib

(guess thats ignorable:)

Running aclocal -I m4
/usr/share/aclocal/libmcrypt.m4:17: warning: underquoted definition of 
AM_PATH_LIBMCRYPT
/usr/share/aclocal/libmcrypt.m4:17:   run info '(automake)Extending aclocal'
/usr/share/aclocal/libmcrypt.m4:17:   or see 
http://sources.redhat.com/automake/automake.html#Extending-aclocal
Running autoheader

BTW: "make install" fails because of "install-docs" in Makefile,
maybe it would make sense to change the Makefile to be more
aware of missing "dtplite"-missing situations.

Nice: The ns_thread.test(s) double the memory usage, saturate my box with
100% CPU load... but, once done, all falls back to where it started.

cu
BE

Re: [naviserver-devel] Chnaged my mind: switch to mercurial

[Stephen D. <sd...@gm...>]

On 11/20/08, Vasiljevic Zoran <zv...@ar...> wrote:
>
>  On 20.11.2008, at 20:33, Stephen Deasey wrote:
>
>  >
>  > And clean your finger nails, and brush your teeth, and say your
>  > prayers...
>
>
> Eh... brave new world...
>
>  So yet another thing to learn... I guess Internet is full of docs
>  about this and I will have fun reading it.


Hopefully, it will take you about 2 minutes to get the basics, which
is the advantage of mercurial over git, an otherwise great system.

Clone the repository first. You need the bits on your hard drive:


$ hg clone https://zo...@bi.../naviserver/naviserver/
$ cd naviserver

... hack hack hack...

$ hg commit
$ hg push


Now obviously there are bells n' whistles, but the basics are easy
enough that it shouldn't prevent you from getting things done.


(The built-in help is excellent: hg help, hg pull --help, etc.)

Re: [naviserver-devel] Chnaged my mind: switch to mercurial

[Vasiljevic Z. <zv...@ar...>]

On 20.11.2008, at 20:33, Stephen Deasey wrote:

>
> And clean your finger nails, and brush your teeth, and say your  
> prayers...

Eh... brave new world...

So yet another thing to learn... I guess Internet is full of docs
about this and I will have fun reading it.

Cheers,
Zoran

Re: [naviserver-devel] Chnaged my mind: switch to mercurial

[Stephen D. <sd...@gm...>]

On 11/20/08, Vlad Seryakov <vl...@cr...> wrote:
> vseryakov is my username,


OK. I've given you write access.


>  looks oike adding openssh pub key does not work, it keeps saying SSH key is not valid


Perhaps during cut 'n paste a stray \n crept in?  Worked for me.

Anyway, you don't need to use ssh, ssl works just as well (http is
mercurial's native transport):

$ hg clone https://vse...@bi.../naviserver/naviserver/
~/in/naviserver-hg



btw. it seems that the bitbucket site is set up such that all repos
belong to a person, so for the naviserver repo I created a 'fake'
person 'naviserver' to own it. I'll send you and Zoran the password,
in case I'm run over by a walrus or something.

You won't need to use it on a day to day basis. It's just for adding
more people to the commit list and changing the details on the
website.


Also, make sure you have this in your ~/.hgrc

[ui]
username=Vlad Seryakov <vse...@cr...>

so that it matches all the other log entries, otherwise you'll be
vlad@localhost or whatever.

And for log messages note that the convention is to have a single
line, like an email subject, then a blank line, then any extra
explanation, in normal paragraphs. Word wrap to 70-80 characters. Use
active voice, 'fix' rather than 'fixed'. No full stop. if it's a
module, prefix with 'nsperm: ' or whatever. Check what's there
already, you'll get the idea. And because commit and push are
separate, you can check and redo as many times as you like before
anything is public.

If you find yourself trying to describe two things, they should have
been separate commits. Mercurial makes this easy. You can commit
multiple times, and then at the end push to the public repo.

And clean your finger nails, and brush your teeth, and say your prayers...

Re: [naviserver-devel] Chnaged my mind: switch to mercurial

[Vlad S. <vl...@cr...>]

vseryakov is my username,

looks oike adding openssh pub key does not work, it keeps saying SSH key is not valid

Stephen Deasey wrote:
> On Wed, Aug 20, 2008 at 7:16 AM, Vasiljevic Zoran <zv...@ar...> wrote:
>> On 20.08.2008, at 00:19, Stephen Deasey wrote:
>>
>>> I think it's probably better to stay with the familiar cvs. It doesn't
>>> scare anyone, SF handles everything, and patches are probably too
>>> infrequent to for the vcs to make much difference anyway.
>>>
>>> Agree?
>> Yes. Lets keep it simple for now.
>>
> 
> 
> I've changed my mind about this again. It's just too much of a pain in
> the ass to do anything more than minor fixes without trampling on
> other peoples stuff, using CVS.
> 
> Here's something new though:
> 
>   http://www.bitbucket.org/naviserver/naviserver/overview/
> 
> Fancy hosting for mercurial repos. It's like github (if you're
> familiar with that, very popular), but for mercurial. It's up to date
> for both branches of AOLserver, and naviserver with the exception of
> Zoran's commit the other day (I'll update that now).  It also models
> the fact that naviserver is a fork AOLserver 4.0.10.
> 
> This would have all the advantages we talked about previously, plus a
> couple of new ones which would be particularly helpful when developing
> larger changes which need a bit of back and forth development before
> being committed: branches a patch queues.
> 
> If you look at the interface you'll see a 'fork' link. The idea is any
> random stranger can fork any project to make changes. They then send a
> pull request to you and merge the changes back, and they delete their
> fork.  Bitbucket also supports mercurial queues, which is almost the
> same but a bit easier when reworking a set of patches.
> 
> So the ideas is that we'd use bitbucket for hosting the mercurial
> repos and keep everything else as is.
> 
> How does this sound?
> 
> 
> (Tell me you username on bitbucket and I'll add you to the project).
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] make memcheck / valgrind

[Stephen D. <sd...@gm...>]

On Thu, Nov 20, 2008 at 8:46 AM, Bernd Eidenschink <eid...@we...> wrote:
>> > ==3781== ERROR SUMMARY: 18 errors from 9 contexts (suppressed: 106 from
>> > 1)
>>
>> This is the important bit. There should be no errors. Scroll back up
>> the log output and you should see details of the errors. Unlike the
>> summary, valgrind outputs these as they happen.
>>
>> I haven't tested against 8.5 in a long time, so I just tried it.
>
> Compiling 8.5 was just out of curiosity. I memchecked both 8.5 and 8.4 (latest
> sources) on naviserver trunk code and had errors with both.
>
> Doesn't matter to me as I just need a running server (and the initial impulse
> for make memcheck was just my test with vtmalloc.).
> I was just curious if the result is somewhat normal, as memcheck runs all the
> tests and one or more of them might intentionally peek and poke.
> (I didn't do one single click, just watched what make memcheck runs until it
> presents the results).
>
> So the result: Using trunk naviserver source code, using latest TCL 8.4 source
> code, compiling and memchecking spits out errors that can be ignored, unless
> barking in uppercase letters or freezing the os :-)
>

No no. Errors are bad, there should be none.

Stop teasing and show us them...   :-)


( Make sure you compile Tcl and naviserver with --enable-symbols )

Re: [naviserver-devel] Chnaged my mind: switch to mercurial

[Vlad S. <vl...@cr...>]

I frankly don't care about any particular system, it just should be only one, fast and accessible all the time.

Stephen Deasey wrote:
> On Wed, Aug 20, 2008 at 7:16 AM, Vasiljevic Zoran <zv...@ar...> wrote:
>> On 20.08.2008, at 00:19, Stephen Deasey wrote:
>>
>>> I think it's probably better to stay with the familiar cvs. It doesn't
>>> scare anyone, SF handles everything, and patches are probably too
>>> infrequent to for the vcs to make much difference anyway.
>>>
>>> Agree?
>> Yes. Lets keep it simple for now.
>>
> 
> 
> I've changed my mind about this again. It's just too much of a pain in
> the ass to do anything more than minor fixes without trampling on
> other peoples stuff, using CVS.
> 
> Here's something new though:
> 
>   http://www.bitbucket.org/naviserver/naviserver/overview/
> 
> Fancy hosting for mercurial repos. It's like github (if you're
> familiar with that, very popular), but for mercurial. It's up to date
> for both branches of AOLserver, and naviserver with the exception of
> Zoran's commit the other day (I'll update that now).  It also models
> the fact that naviserver is a fork AOLserver 4.0.10.
> 
> This would have all the advantages we talked about previously, plus a
> couple of new ones which would be particularly helpful when developing
> larger changes which need a bit of back and forth development before
> being committed: branches a patch queues.
> 
> If you look at the interface you'll see a 'fork' link. The idea is any
> random stranger can fork any project to make changes. They then send a
> pull request to you and merge the changes back, and they delete their
> fork.  Bitbucket also supports mercurial queues, which is almost the
> same but a bit easier when reworking a set of patches.
> 
> So the ideas is that we'd use bitbucket for hosting the mercurial
> repos and keep everything else as is.
> 
> How does this sound?
> 
> 
> (Tell me you username on bitbucket and I'll add you to the project).
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] make memcheck / valgrind

[Bernd E. <eid...@we...>]

> > ==3781== ERROR SUMMARY: 18 errors from 9 contexts (suppressed: 106 from
> > 1)
>
> This is the important bit. There should be no errors. Scroll back up
> the log output and you should see details of the errors. Unlike the
> summary, valgrind outputs these as they happen.
>
> I haven't tested against 8.5 in a long time, so I just tried it.

Compiling 8.5 was just out of curiosity. I memchecked both 8.5 and 8.4 (latest 
sources) on naviserver trunk code and had errors with both.

Doesn't matter to me as I just need a running server (and the initial impulse 
for make memcheck was just my test with vtmalloc.).
I was just curious if the result is somewhat normal, as memcheck runs all the 
tests and one or more of them might intentionally peek and poke.
(I didn't do one single click, just watched what make memcheck runs until it 
presents the results).

So the result: Using trunk naviserver source code, using latest TCL 8.4 source 
code, compiling and memchecking spits out errors that can be ignored, unless 
barking in uppercase letters or freezing the os :-)

cu
BE

Re: [naviserver-devel] Changed my mind: switch to mercurial

[Stephen D. <sd...@gm...>]

On Wed, Nov 19, 2008 at 6:54 PM, Andrew Piskorski <at...@pi...> wrote:
> On Wed, Nov 19, 2008 at 05:48:11PM +0000, Stephen Deasey wrote:
>
>>   http://www.bitbucket.org/naviserver/naviserver/overview/
>
>> It's up to date for both branches of AOLserver, and naviserver with
>> the exception of Zoran's commit the other day (I'll update that
>> now).  It also models the fact that naviserver is a fork AOLserver
>> 4.0.10.
>
> I've yet to really use Mercurial at all, but that all sounds very
> cool.
>
> I take it that one way CVS to Mercurial importing is very robust by
> now?


It is. Here's the command I use:

$ hg convert -A ~/authors.map \
      --config convert.hg.usebranchnames=0 --config convert.hg.clonebranches=1 \
      ~/in/naviserver-HEAD ns-HEAD-hg

Where authors.map is a file which maps sf username like 'sdeasey' to
mercurial users like 'Stephen Deasey <sd...@gm...>', and the
branch stuff is saying I prefer to represent branches as separate
repos.


However, I have done a bunch of clean ups of the commit messages, some
squashing of commits which should have been one commit, or where a
followup commit fixed a typo or some other trivial mistake with a log
message of 'oops...'.

So the conversion is 'accurate', but better than the original :-)

Even if you're not interested in mercurial, you might want to browse
the aolserver repos here for this reason alone:

  http://www.bitbucket.org/aolserver/aolserver/overview/
  http://www.bitbucket.org/aolserver/aolserver-40x/overview/

For example, according to the AOLserver ChangeLog the last change was
made in June, and it's one of only two changes made this year:

  http://aolserver.cvs.sourceforge.net/viewvc/aolserver/aolserver/ChangeLog?revision=1.386&view=markup

Browsing the mercurial shortlog on bitbucket you can see there's
actually been 9 changes by 4 different authors.

This would be useful for these guys:

  http://openacs.org/forums/message-view?message_id=2438982

to whom it looks, on the outside, like nothing's been touched in
either aolserver or naviserver for years...

Re: [naviserver-devel] Changed my mind: switch to mercurial

[Andrew P. <at...@pi...>]

On Wed, Nov 19, 2008 at 05:48:11PM +0000, Stephen Deasey wrote:

>   http://www.bitbucket.org/naviserver/naviserver/overview/

> It's up to date for both branches of AOLserver, and naviserver with
> the exception of Zoran's commit the other day (I'll update that
> now).  It also models the fact that naviserver is a fork AOLserver
> 4.0.10.

I've yet to really use Mercurial at all, but that all sounds very
cool.

I take it that one way CVS to Mercurial importing is very robust by
now?

-- 
Andrew Piskorski <at...@pi...>
http://www.piskorski.com/

[naviserver-devel] Chnaged my mind: switch to mercurial

[Stephen D. <sd...@gm...>]

On Wed, Aug 20, 2008 at 7:16 AM, Vasiljevic Zoran <zv...@ar...> wrote:
>
> On 20.08.2008, at 00:19, Stephen Deasey wrote:
>
>> I think it's probably better to stay with the familiar cvs. It doesn't
>> scare anyone, SF handles everything, and patches are probably too
>> infrequent to for the vcs to make much difference anyway.
>>
>> Agree?
>
> Yes. Lets keep it simple for now.
>


I've changed my mind about this again. It's just too much of a pain in
the ass to do anything more than minor fixes without trampling on
other peoples stuff, using CVS.

Here's something new though:

  http://www.bitbucket.org/naviserver/naviserver/overview/

Fancy hosting for mercurial repos. It's like github (if you're
familiar with that, very popular), but for mercurial. It's up to date
for both branches of AOLserver, and naviserver with the exception of
Zoran's commit the other day (I'll update that now).  It also models
the fact that naviserver is a fork AOLserver 4.0.10.

This would have all the advantages we talked about previously, plus a
couple of new ones which would be particularly helpful when developing
larger changes which need a bit of back and forth development before
being committed: branches a patch queues.

If you look at the interface you'll see a 'fork' link. The idea is any
random stranger can fork any project to make changes. They then send a
pull request to you and merge the changes back, and they delete their
fork.  Bitbucket also supports mercurial queues, which is almost the
same but a bit easier when reworking a set of patches.

So the ideas is that we'd use bitbucket for hosting the mercurial
repos and keep everything else as is.

How does this sound?


(Tell me you username on bitbucket and I'll add you to the project).

Re: [naviserver-devel] make memcheck / valgrind

[Stephen D. <sd...@gm...>]

On Wed, Nov 19, 2008 at 3:45 PM, Bernd Eidenschink <eid...@we...> wrote:
>> > But the results on my local box are... a little bit scary right now.
>>
>> ... scary is relative. So, how scary is your "scary", really?
>
> Scary for me, the ignorant :-)
>
>
> ==3781== ERROR SUMMARY: 18 errors from 9 contexts (suppressed: 106 from 1)


This is the important bit. There should be no errors. Scroll back up
the log output and you should see details of the errors. Unlike the
summary, valgrind outputs these as they happen.

I haven't tested against 8.5 in a long time, so I just tried it.

I always compile Tcl for testing with --enable-symbols=mem, which you
would think would interact badly with valgrind, but I've found it's
like a pig sniffing out truffles when it comes to buffer over runs and
depending on uninitialized memory. These are the kinds of errors
valgrind is talking about, in contrast to plain memory leaks which you
might expect to find.

Anyway, won't even run with 8.5.5 here:

[19/Nov/2008:16:48:33][26547.b7ff46c0][-main-] Fatal: expected to
create new entry for object map

Program received signal SIGABRT, Aborted.
0x00110416 in __kernel_vsyscall ()
Missing separate debuginfos, use: debuginfo-install gcc.i386 glibc.i686
(gdb) bt
#0  0x00110416 in __kernel_vsyscall ()
#1  0x00b42660 in raise () from /lib/libc.so.6
#2  0x00b44028 in abort () from /lib/libc.so.6
#3  0x00144f18 in Panic (fmt=0x2b0760 "expected to create new entry
for object map") at log.c:617
#4  0x0025fec7 in Tcl_PanicVA (format=0x2b0760 "expected to create new
entry for object map",
    argList=0xbfc06694 "") at
/home/sd/src/tcl8.5.5/unix/../generic/tclPanic.c:93
#5  0x0025ffca in Tcl_Panic (format=0x2b0760 "expected to create new
entry for object map")
    at /home/sd/src/tcl8.5.5/unix/../generic/tclPanic.c:132
#6  0x0025c7b6 in TclDbInitNewObj (objPtr=0x96b7220) at
/home/sd/src/tcl8.5.5/unix/../generic/tclObj.c:637
#7  0x0024b919 in Tcl_DbNewListObj (objc=1, objv=0xbfc06700,
    file=0x2ac234
"/home/sd/src/tcl8.5.5/unix/../generic/tclFileName.c", line=787)
    at /home/sd/src/tcl8.5.5/unix/../generic/tclListObj.c:239
#8  0x00227d72 in Tcl_FSJoinToPath (pathPtr=0x96e4b50, objc=0, objv=0x0)
    at /home/sd/src/tcl8.5.5/unix/../generic/tclFileName.c:787
#9  0x0026639d in SetFsPathFromAny (interp=0x0, pathPtr=0x96e4b50)
    at /home/sd/src/tcl8.5.5/unix/../generic/tclPathObj.c:2441
#10 0x00265e4b in TclFSSetPathDetails (pathPtr=0x96e4b50,
fsRecPtr=0x96b4640, clientData=0x0)
    at /home/sd/src/tcl8.5.5/unix/../generic/tclPathObj.c:2203
#11 0x0024a1c3 in Tcl_FSGetFileSystemForPath (pathPtr=0x96e4b50)
    at /home/sd/src/tcl8.5.5/unix/../generic/tclIOUtil.c:4437
#12 0x00248a63 in Tcl_FSGetCwd (interp=0x0) at
/home/sd/src/tcl8.5.5/unix/../generic/tclIOUtil.c:2736
#13 0x00148603 in SetCwd (path=0x96f53c8
"/home/sd/ns-scratch-hg/tests") at nsmain.c:1063
#14 0x00147c24 in Ns_Main (argc=8, argv=0xbfc06b74, initProc=0x8048636
<ServerInit>) at nsmain.c:504
#15 0x0804862c in main (argc=Cannot access memory at address 0x67b3
) at main.c:64


Seems innocuous enough:

nsd/nsmain.c:

char *
SetCwd(char *path)
{
    Tcl_Obj *pathObj;

    pathObj = Tcl_NewStringObj(path, -1);
    Tcl_IncrRefCount(pathObj);
    if (Tcl_FSChdir(pathObj) == -1) {
        Ns_Fatal("nsmain: chdir(%s) failed: '%s'", path,
strerror(Tcl_GetErrno()));
    }
    Tcl_DecrRefCount(pathObj);
    pathObj = Tcl_FSGetCwd(NULL);
    if (pathObj == NULL) {
        Ns_Fatal("nsmain: can't resolve home directory path");
    }

    return (char *)Tcl_FSGetTranslatedStringPath(NULL, pathObj);
}


Is this a bug though?  Can Tcl really expect to have never seen this
pointer before:

tcl8.5.5/generic/tclObj.c:

void
TclDbInitNewObj(
    register Tcl_Obj *objPtr)
{
    objPtr->refCount = 0;
    objPtr->bytes = tclEmptyStringRep;
    objPtr->length = 0;
    objPtr->typePtr = NULL;

#ifdef TCL_THREADS
    /*
     * Add entry to a thread local map used to check if a Tcl_Obj was
     * allocated by the currently executing thread.
     */

    if (!TclInExit()) {
        Tcl_HashEntry *hPtr;
        Tcl_HashTable *tablePtr;
        int isNew;
        ThreadSpecificData *tsdPtr = TCL_TSD_INIT(&dataKey);

        if (tsdPtr->objThreadMap == NULL) {
            tsdPtr->objThreadMap = (Tcl_HashTable *)
                    ckalloc(sizeof(Tcl_HashTable));
            Tcl_InitHashTable(tsdPtr->objThreadMap, TCL_ONE_WORD_KEYS);
        }
        tablePtr = tsdPtr->objThreadMap;
        hPtr = Tcl_CreateHashEntry(tablePtr, (char *) objPtr, &isNew);
        if (!isNew) {
            Tcl_Panic("expected to create new entry for object map");
        }
        Tcl_SetHashValue(hPtr, NULL);
    }
#endif /* TCL_THREADS */
}

Re: [naviserver-devel] make memcheck / valgrind

[Vasiljevic Z. <zv...@ar...>]

On 19.11.2008, at 16:45, Bernd Eidenschink wrote:

> ==3781== LEAK SUMMARY:
> ==3781==    definitely lost: 72 bytes in 5 blocks.
> ==3781==    indirectly lost: 120 bytes in 10 blocks.
> ==3781==      possibly lost: 139,672,920 bytes in 2,504 blocks.
> ==3781==    still reachable: 8,944,636 bytes in 1,508 blocks.
> ==3781==         suppressed: 0 bytes in 0 blocks.
>


Valgrind manual tells:

"possibly lost" means your program is probably leaking memory,
     -> unless you're doing funny things with pointers <-

(Other memtools like Purify do the same)

So we have few real leaks. Most of it is due to some external
extension that allocates something, then points into the alocated
block and then releases the memory. This could mean a leak/growth
but must not be. To find if this is true, kick "top" and observe
how the process virtual size behaves when you repeatedly hit your
pages (or use ab to simulate load). If this grows obviously and
increasingly as the requesst commence, then you have a leak.

I would say that naviserver is reasonably leak-free under most
circumstances, Tcl 8.4 as well. I can also speak for some extensions
like Tdom, Xotcl. We use them *extensively* and have absolutely
no memory issues (ok, perhaps we do, but not like the above).

Re: [naviserver-devel] make memcheck / valgrind

[Bernd E. <eid...@we...>]

> > But the results on my local box are... a little bit scary right now.
>
> ... scary is relative. So, how scary is your "scary", really?

Scary for me, the ignorant :-)


==3781== ERROR SUMMARY: 18 errors from 9 contexts (suppressed: 106 from 1)
==3781== malloc/free: in use at exit: 148,617,748 bytes in 4,027 blocks.
==3781== malloc/free: 6,481 allocs, 2,454 frees, 183,008,167 bytes allocated.
==3781== For counts of detected errors, rerun with: -v
==3781== searching for pointers to 4,027 not-freed blocks.
==3781== checked 258,234,212 bytes.

==3781== LEAK SUMMARY:
==3781==    definitely lost: 72 bytes in 5 blocks.
==3781==    indirectly lost: 120 bytes in 10 blocks.
==3781==      possibly lost: 139,672,920 bytes in 2,504 blocks.
==3781==    still reachable: 8,944,636 bytes in 1,508 blocks.
==3781==         suppressed: 0 bytes in 0 blocks.

(this is the non-vtmalloc-default result)

> (BTW, if I were you, I would turn ANY "clever" memory allocator and use
> malloc/free everywhere when debugging memory.)

My idea was: Give vtmalloc a try (as freeing memory always is nice, we have 
lots of memory intense XML import stuff to do) - but as I've never used it 
before, i ran into this appealing "memcheck" option ...

> As from our experience... if we leak (which is not unlikely) then
> it is far from scary, as otherwise I'd already pull-out all my hair.
> However, when looking in the mirror, I can say we are not leaking.

There is no correlation, believe me: We ware not leaking also, but our 
hairs... :-)

cu
BE

Re: [naviserver-devel] make memcheck / valgrind

[Vasiljevic Z. <zv...@ar...>]

On 19.11.2008, at 15:13, Bernd Eidenschink wrote:

> I compiled the trunk against TCL 8.5.5, the first time with  
> "vtmalloc", the
> second time with original untouched Tcl.
>

Cannot comment on 8.5.5. Still using 8.4 branch (conservative user- 
base, heh).
But...

> But the results on my local box are... a little bit scary right now.
>

... scary is relative. So, how scary is your "scary", really?

(BTW, if I were you, I would turn ANY "clever" memory allocator and use
malloc/free everywhere when debugging memory.)

As from our experience... if we leak (which is not unlikely) then
it is far from scary, as otherwise I'd already pull-out all my hair.
However, when looking in the mirror, I can say we are not leaking.

Cheers
Zoran

[naviserver-devel] make memcheck / valgrind

[Bernd E. <eid...@we...>]

Hey ya,

just a quick question:

Is the result of "make memcheck" - currently - supposed to return "0 errors " 
in the error summary, likewise "0 bytes definitely lost" in the leak summary?

I compiled the trunk against TCL 8.5.5, the first time with "vtmalloc", the 
second time with original untouched Tcl.

But the results on my local box are... a little bit scary right now.

Bernd.

Re: [naviserver-devel] PHP streaming hangs

[Vlad S. <vl...@cr...>]

What distribution do you use? In my Archlinux this does not work, they 
provide Apache module only, so i need to recompile PHP anyway.

In such cases README and automated way to rebuild it would be nice, it 
is gone from Makefile now

Stephen Deasey wrote:
> On Sat, Nov 8, 2008 at 8:47 PM, Vlad Seryakov <vl...@cr...> wrote:
>>> Do you need to buy into the full build process? Maybe you can just
>>> pull in the header and link against the library..?
>> The problem with PHP that it can support only one SAPI implementation
>> and nsphp is not just extesion but SAPI handler, so just compiling it as
>> extension will not work
>>
> 
> 
> There's an --enable-embed=shared option, and it seems to work (added).
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] naviserver ChangeLog,1.827,1.828

[Vlad S. <vl...@cr...>]

Not sure what are you asking?

Stephen Deasey wrote:
> On Sat, Nov 8, 2008 at 10:42 PM, Vlad Seryakov
> <ser...@us...> wrote:
>> Update of /cvsroot/naviserver/naviserver
>> In directory fdv4jf1.ch3.sourceforge.com:/tmp/cvs-serv26902
>>
>> Modified Files:
>>        ChangeLog
>> Log Message:
>> mark connection flags when last chunk is sent
>>
>>
>> Index: ChangeLog
>> ===================================================================
>> RCS file: /cvsroot/naviserver/naviserver/ChangeLog,v
>> retrieving revision 1.827
>> retrieving revision 1.828
>> diff -C2 -d -r1.827 -r1.828
>> *** ChangeLog   7 Nov 2008 19:01:57 -0000       1.827
>> --- ChangeLog   8 Nov 2008 22:42:09 -0000       1.828
>> ***************
>> *** 1,2 ****
>> --- 1,7 ----
>> + 2008-11-08  Vlad Seryakov <ser...@us...>
>> +
>> +       * nsd/connio.c: Mark connection flags with NS_CONN_SENT_LAST_CHUNK
>> +       when last chunk is actuall sent.
>> +
> 
> 
> What?
> 
> 
> $ grep -r NS_CONN_SENT_LAST_CHUNK *
> 
> ChangeLog:   * nsd/connio.c: Mark connection flags with NS_CONN_SENT_LAST_CHUNK
> include/ns.h:   #define NS_CONN_SENT_LAST_CHUNK    0x100 /* Marks that
> the last chunk was sent in chunked mode */
> nsd/connio.c:  connPtr->flags |= NS_CONN_SENT_LAST_CHUNK;
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
> 

Re: [naviserver-devel] PHP streaming hangs

[Stephen D. <sd...@gm...>]

On Sat, Nov 8, 2008 at 8:47 PM, Vlad Seryakov <vl...@cr...> wrote:
>>
>> Do you need to buy into the full build process? Maybe you can just
>> pull in the header and link against the library..?
>
> The problem with PHP that it can support only one SAPI implementation
> and nsphp is not just extesion but SAPI handler, so just compiling it as
> extension will not work
>


There's an --enable-embed=shared option, and it seems to work (added).

Re: [naviserver-devel] naviserver ChangeLog,1.827,1.828

[Stephen D. <sd...@gm...>]

On Sat, Nov 8, 2008 at 10:42 PM, Vlad Seryakov
<ser...@us...> wrote:
> Update of /cvsroot/naviserver/naviserver
> In directory fdv4jf1.ch3.sourceforge.com:/tmp/cvs-serv26902
>
> Modified Files:
>        ChangeLog
> Log Message:
> mark connection flags when last chunk is sent
>
>
> Index: ChangeLog
> ===================================================================
> RCS file: /cvsroot/naviserver/naviserver/ChangeLog,v
> retrieving revision 1.827
> retrieving revision 1.828
> diff -C2 -d -r1.827 -r1.828
> *** ChangeLog   7 Nov 2008 19:01:57 -0000       1.827
> --- ChangeLog   8 Nov 2008 22:42:09 -0000       1.828
> ***************
> *** 1,2 ****
> --- 1,7 ----
> + 2008-11-08  Vlad Seryakov <ser...@us...>
> +
> +       * nsd/connio.c: Mark connection flags with NS_CONN_SENT_LAST_CHUNK
> +       when last chunk is actuall sent.
> +


What?


$ grep -r NS_CONN_SENT_LAST_CHUNK *

ChangeLog:   * nsd/connio.c: Mark connection flags with NS_CONN_SENT_LAST_CHUNK
include/ns.h:   #define NS_CONN_SENT_LAST_CHUNK    0x100 /* Marks that
the last chunk was sent in chunked mode */
nsd/connio.c:  connPtr->flags |= NS_CONN_SENT_LAST_CHUNK;

Re: [naviserver-devel] PHP streaming hangs

[Vlad S. <vl...@cr...>]

The major difference between previous version and current that in the 
previous version we always closed connection, even in 1.1, PHP streaming 
was not using chunked encoding explicitly and the server never 
auto-assigned it.
In current version the server decided and because we always keep-alive 
and in chunked, Firefox after 3 request hangs because reply is not 
correct. This is strange because in telnet session it looks fine but my 
eye may not be protocol-demanding like Firefox.

Vlad Seryakov wrote:
>> Does this actually need to flush?
>>
>> It depends when php calls it. In the case where there's no body, eg. a
>> 302 redirect, then maybe this signals 'all done' and you're to write
>> the headers. In the case where there is a body it's not needed because
>> naviserver will construct and dump the headers when you try to write
>> the first data, and it will send both with a single syscall (and
>> packet, if it fits), which is more efficient.
>>
>> So you should try not to flush. You might try to do nothing in this
>> call, and detect when you've finished sending in the case of a
>> zero-length body some other way. When php returns control to your
>> Ns_Op function you could check to see if any bytes were sent and if
>> not, flush the headers.
> 
> I tried it, with code before senfile implementation where we had flush 
> function it worked perfectly, so PHP is calling this only when it needs 
> to actually send headers. Not sure i can find out about body coming, 
> even in 302 case there can be body but may be not. Will try again
> 
>> But anyway, you're checking for ctx->conn != NULL before setting the
>> response code, then using it unconditionally when you write the data.
>>
>> Ns_ConnWriteData can fail but you're ignoring the return code and
>> returning success
> 
> It is mostly possible that the problem is in nsphp :-((
> 
>>
>> Does php try to set a length header? You may need to check for it in
>> here and call Ns_ConnSetLengthHeader.
>>
> 
> This i checked, no length
> 
> 
>> Whether the write succeeds or fails, you're returning the original
>> buffer length, which I guess signals success to php. It might be hard
>> to figure out exactly how much did get sent, because
>> conn->nContentSent includes the headers bytes. But maybe php doesn't
>> care? You'll have to check if you can return -1 or 0 or something.
> 
> when abort is called, i do not return str_length, it might be logjmp 
> inside php
> 
> 
>> You might also want to check if php exposes it's buffering. If you can
>> figure out that your ub_write call is being called with all the data
>> that's going to be sent (the script has finished executing) or the
>> last piece of data, you shouldn't pass the flag NS_CONN_STREAM.
>> Otherwise, you should.
> 
> No buffering, PHP calls it with all pieces separately
> 
>>
>>> I tried that but could not get it to work, it generates configure but
>>> when i run it i never could make it finish, there are always some errors.
>>
>>
>> Do you need to buy into the full build process? Maybe you can just
>> pull in the header and link against the library..?
> 
> The problem with PHP that it can support only one SAPI implementation 
> and nsphp is not just extesion but SAPI handler, so just compiling it as 
> extension will not work
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> naviserver-devel mailing list
> nav...@li...
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>