Menu
▾
▴
[Felix-impl] JudySL problem
|
From: john s. <sk...@us...> - 2011-10-29 22:40:57
|
Hi, I'm having some problems with JudySL .. not sure what's going on.
First a test program written in Felix:
/////
open Judy;
x := JSLArray();
var je: JError_t;
var pv : &word;
var buffer: &char = C_hack::cast[&char] (C_hack::malloc(1000));
for var i in 0 upto 100 do
var j = Cstdlib::rand();
strcpy(buffer, j.str.cstr);
println$ str i + " --> " + str j + " = " + buffer;
JudySLIns(x,buffer,&je,&pv);
if C_hack::cast[long] (*pv) != 0l do println "DUPLICATE"; done
*pv = C_hack::cast[word] i;
done
println "Listing ds=d->s";
*buffer=char 0;
JudySLFirst(x,buffer,&je,&pv);
var k= 0;
whilst not isNULL pv do
j := C_hack::cast[int] (*pv);
println$ "Recorded copy " + k.str + ": " + string buffer + " => " + str j;
++k;
JudySLNext(x,buffer,&je,&pv);
done
/////
I hope this is readable even to a C programmer!
What happens: as written this stores only 45 entries.
When I change the count to 1000 it segfaults during the
storage phase.
OK so here is the Felix binding:
proc JudySLIns: JSLArray * &char * &JError_t * &&word =
"""
if (::std::strlen($2) >= JUDY_SL_MAXLEN) throw "JudySLIns strlen>10000";
*(Word_t**)$4=(Word_t*)JudySLIns($1,(unsigned char*)$2,$3);
""";
There's one more thing to look at: Felix now allows JudySL arrays to contain Felix
pointers as values. The GC uses the follow routine to scan a JudySL array
for such values:
void *JudySL_scanner(collector_t *collector, gc_shape_t *shape, void *pp, unsigned long dyncount, int reclimit)
{
void *p = *(void**)pp;
//printf("Scanning judyL array %p->%p\n", pp, p);
JError_t je;
unsigned char *key = (unsigned char*)::std::malloc(10000); // HACK
*key = 0;
Word_t *pval = 0;
pval = (Word_t*)JudySLFirst(p, key, &je);
while(pval) {
//printf("JudyL scanning p=%s\n",key);
collector->register_pointer((void*)*pval,reclimit);
pval = (Word_t*)JudySLNext(p, key, &je);
}
::std::free(key);
return 0;
}
The collector->register_pointer() function simply checks if the value is a Felix
pointer, if it is, the pointer is "marked" by the GC as reachable and not to be
collected. It is unlikely this code is invoked! I uncommented the diagostic and
it isn't being printed but I include this code for completeness.
The output of the Felix program with loop counter 1000 shows:
779 --> 1728536152 = 1728536152
780 --> 348330048 = 348330048
DUPLICATE
781 --> 342695014 = 342695014
....
816 --> 1595850963 = 1595850963
817 --> 1543867758 = 1543867758
818 --> 1887985652 = 1887985652
Segmentation fault
A duplicate is not likely (pseudo random number generators do not duplicate).
The program "flx_cp" is a sane replacement for unix "cp". This program stores
the source and destination of a wildcarded filecopy command in two judy SL arrays
(src->dst and dst->src) so it can check for overlap before copying. The index is a
null terminated string as required by JudySL, the value is a pointer to a C++ string.
In that program the index values are scrambled exactly as if the keys are overwritting.
It is possible that Erick has screwed up the Judy build. Judy used to be built by the
standard Felix build code, but because of code generation in Judy Makefile
and huge ugly macros to reuse code, I reorganised the code to eliminate the code
generation (by pre-generating the required tables) and copied some files about
to eliminate the need for complex include directives. The generated code is actually
a special indexing table for either 32 or 64 bit Judy. In fact there's no reason to generate
this code since there are exactly two options.. and there even no reason not to build
Judy for BOTH 32 and 64 bit (it can work with BOTH sizes on both 32 and 64 bit machines
with a bit of hacking about to decouple the binary size used for the data structure from
the C types used to manipulate them).
So it is possible the build is screwed up. However I think Judy1 and JudyL work
since they're used extensively by the GC (although .. if they don't it would explain
why some programs crash in the GC .. such as the webserver .. but it seems more
likely to be a bug in the GC).
Bottom line: it looks like a bug in JudySL but Judy has proven to be bug free if built correctly,
just hard to use properly. So it is more likely a bug in my code .. I just can't see it.
On 30/10/2011, at 5:33 AM, Jason E. Aten wrote:
> did you get the JudySLIns figured out?
>
> Here's how I wrap the JLSI macro... which is just
FYI: as you can see by my code I do not use the macros. The confuse values
with lvalues. The function calls make the distinction explicit. Unfortunately it is
the macros that are documented so I have to backtrack the documentation
from the function calls (an extra cognitive layer) and then also consider
the Felix binding .. as well as checking proper use in my application.
>
> void insertkey(const char* key, V value)
> {
>
> PWord_t PValue = 0;
> uint8_t Index[BUFSIZ]; // string to sort.
> strncpy((char*)Index,key,BUFSIZ);
>
> JSLI(
> PValue,
> (_judyS),
> Index); // store string into array
> if (0==PValue) {
> assert(0);
> }
> if ((long)PValue == -1) {
> assert(0);
> }
>
> // was here to diagnose duplicates t_typ's
> // but we never want to overwrite old string without free-ing
> them first,
> // so leave this insert in.
> if (*PValue != 0) {
> assert(0);
> }
>
> ++_size;
> (*((V*)PValue)) = value;
> }
>
>
>
> On Sat, Oct 29, 2011 at 9:21 AM, john skaller
> <sk...@us...> wrote:
>> flx_cp isn't working .. it seems I;m not using JudySLIns correctly:
>>
>> tools/flx_cp web "(.*)" 'xxx\1' --test
>>
>> ////////
>>
>> Consider file download.html
>> Listing ds=d->s
>> Listing sd=s->d
>> subs 1 -> download.html
>> [pre-Ins] Src = web/download.html Dst= xxxdownload.html
>> [post-Ins sd] Src = web/download.html Dst= xxxdownload.html
>> [post-Ins ds] Src = web/download.html Dst= xxxdownload.html
>> Copy web/download.html -> xxxdownload.html
>> Consider file Easy_to_Deploy.fdoc
>> Listing ds=d->s
>> Recorded copy xxxdownload.html <- web/download.html
>> Listing sd=s->d
>> Recorded copy web/download.html -> xxxdownload.html
>> subs 1 -> Easy_to_Deploy.fdoc
>> [pre-Ins] Src = web/Easy_to_Deploy.fdoc Dst= xxxEasy_to_Deploy.fdoc
>> [post-Ins sd] Src = web/Easy_to_Deploy.fdoc Dst= xxxEasy_to_Deploy.fdoc
>> [post-Ins ds] Src = web/Easy_to_Deploy.fdoc Dst= xxxEasy_to_Deploy.fdoc
>> Copy web/Easy_to_Deploy.fdoc -> xxxEasy_to_Deploy.fdoc
>> Consider file Easy_to_Read_and_Write.fdoc
>> Listing ds=d->s
>> Recorded copy xxxEasy_to_Deploy.fdoc <- web/Easy_to_Deploy.fdoc
>> Recorded copy xxxdasy_oad.html <- web/download.html
>> Listing sd=s->d
>> Recorded copy web/asy__to_htmloy.fdoc -> xxxEasy_to_Deploy.fdoc
>> Recorded copy web/asy_loadhtmll -> xxxdownload.html
>> subs 1 -> Easy_to_Read_and_Write.fdoc
>> [pre-Ins] Src = web/Easy_to_Read_and_Write.fdoc Dst= xxxEasy_to_Read_and_Write.fdoc
>> [post-Ins sd] Src = web/Easy_to_Read_and_Write.fdoc Dst= xxxEasy_to_Read_and_Write.fdoc
>> [post-Ins ds] Src = web/Easy_to_Read_and_Write.fdoc Dst= xxxEasy_to_Read_and_Write.fdoc
>> Copy web/Easy_to_Read_and_Write.fdoc -> xxxEasy_to_Read_and_Write.fdoc
>> Consider file Easy_to_Read_and_Write_01.cpp
>> Listing ds=d->s
>> Recorded copy xxxEasy_to_Dead_y.fdoc <- web/Easy_to_Deploy.fdoc
>> Recorded copy xxxEasy_to_Read_and_Write.fdoc <- web/Easy_to_Read_and_Write.fdoc
>> Recorded copy xxxdasy_oad.html <- web/download.html
>> Listing sd=s->d
>> Recorded copy web/asy__to_html_andWritte.fdoc -> xxxEasy_to_Read_and_Write.fdoc
>> Segmentation fault
>> ///////////
>>
>> clearly some overwiting happening ....
>>
>>
>> --
>> john skaller
>> sk...@us...
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups "Felix Language" group.
>> To post to this group, send email to fel...@go....
>> To unsubscribe from this group, send email to fel...@go....
>> For more options, visit this group at http://groups.google.com/group/felix-language?hl=en.
>>
>>
--
john skaller
sk...@us...
|
