rest2web-develop

[Rest2web-develop] Re: File Bug

[Fuzzyman <fuz...@vo...>]

Hello Andrew,

I would certainly appreciate your eyes in tracking down this bug. I
don't *think* it can be platform specific though, as there is no
platform specific code (I use platform independent functions from
``os.path`` and ``shutil`` to do all the work).

I can see where the bug is happening - for you it seems to be where the
location of the source file is determined. I've pasted the full code in
below - and explained the logic of the first part which determines the
source location. You could add in a print statement to see where it
thinks the source file *should be*, which will give us a clue as to what
is happening.

The code exists twice (which means it ought to be turned into a method
of Processor). Once in the ``process`` method and once in the
``processindex`` method.

When rest2web starts, it changes the current directory to be the 'start'
directory (top level of the site directory tree).

As it goes through the files, the current directory being worked on is
stored in ``self.dir`` (self being the processor instance). At the top
level directory the following is set in ``__init__``, ``self.dir =
os.curdir``. (Is it possible that this is the cause of the problem ?)

Following code taken from -
http://rest2web.python-hosting.com/file/trunk/rest2web/restprocessor.py
- from the process method.

# also copy files (from restindex) into same dir as target
for entry in restindex['file']:
    src = join(self.dir, dirname(filename), entry)
    ## XXX Add a ``print src`` here :-)
    if not os.path.isfile(src):
        print ('File "%s" referenced by file keyword missing.' %
            entry)
        continue
    #
    target_file_name = url2pathname(target)
    targetfile = os.path.normpath(join(
        self.target, self.dir, target_file_name))
    dest = join(dirname(targetfile), split(entry)[1])
    comparefile = os.path.normpath(join(
        self.compare, self.dir, target_file_name))
    cmp = join(dirname(comparefile), split(entry)[1])
    if comparefiles(src, cmp):
        print 'File "%s" identical, skipping.' % entry
        continue
    if not isdir(dirname(dest)):
        os.makedirs(dirname(dest))
    print 'Copying "%s".' % entry
    copy2(src, dest)   # alias for shutil.copy2

I'll step through the code to explain the process.

    for entry in restindex['file']:

restindex['file'] should be a list of all your file keywords from the
page being processed.


    src = join(self.dir, dirname(filename), entry)

``join`` is an alias for ``os.path.join``.

``filename`` is the name of the file being processed. When we are
processing an index file, it could actually be a path relative to
``self.dir`` if the ``index-file`` option is set.

You were getting lots of  ``File "..." referenced by file keyword
missing.`` errors - so this logic seems to be faulty ?

If you add the print statement I suggest, then you should be able to
tell what is happening. I'm sure the solution is simple enough - but I
can't see it.

Thanks

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

[Rest2web-develop] RE: File Bug

[Andrew I. <and...@us...>]

I love one-line fixes.

Changed
     src = join(self.dir, filename, entry)
to
     src = join(self.dir, entry)

It appears that the filename value set the path to
"index.txt/thefilename.ext" instead of just "thefilename.ext".  Once I
removed it, both the index and non-index pages copied files fine (so I did
not need to change the other loop's src line).

Andrew

> -----Original Message-----
> From: Fuzzyman [mailto:fuz...@vo...]
> Sent: Tuesday, April 04, 2006 2:34 AM
> To: res...@li...; Andrew Ittner
> Subject: Re: File Bug
>
>
> Hello Andrew,
>
> I would certainly appreciate your eyes in tracking down this bug. I
> don't *think* it can be platform specific though, as there is no
> platform specific code (I use platform independent functions from
> ``os.path`` and ``shutil`` to do all the work).
>
> I can see where the bug is happening - for you it seems to be where the
> location of the source file is determined. I've pasted the full code in
> below - and explained the logic of the first part which determines the
> source location. You could add in a print statement to see where it
> thinks the source file *should be*, which will give us a clue as to what
> is happening.
>
> The code exists twice (which means it ought to be turned into a method
> of Processor). Once in the ``process`` method and once in the
> ``processindex`` method.
>
> When rest2web starts, it changes the current directory to be the 'start'
> directory (top level of the site directory tree).
>
> As it goes through the files, the current directory being worked on is
> stored in ``self.dir`` (self being the processor instance). At the top
> level directory the following is set in ``__init__``, ``self.dir =
> os.curdir``. (Is it possible that this is the cause of the problem ?)
>
> Following code taken from -
> http://rest2web.python-hosting.com/file/trunk/rest2web/restprocessor.py
> - from the process method.
>
> # also copy files (from restindex) into same dir as target
> for entry in restindex['file']:
>     src = join(self.dir, dirname(filename), entry)
>     ## XXX Add a ``print src`` here :-)
>     if not os.path.isfile(src):
>         print ('File "%s" referenced by file keyword missing.' %
>             entry)
>         continue
>     #
>     target_file_name = url2pathname(target)
>     targetfile = os.path.normpath(join(
>         self.target, self.dir, target_file_name))
>     dest = join(dirname(targetfile), split(entry)[1])
>     comparefile = os.path.normpath(join(
>         self.compare, self.dir, target_file_name))
>     cmp = join(dirname(comparefile), split(entry)[1])
>     if comparefiles(src, cmp):
>         print 'File "%s" identical, skipping.' % entry
>         continue
>     if not isdir(dirname(dest)):
>         os.makedirs(dirname(dest))
>     print 'Copying "%s".' % entry
>     copy2(src, dest)   # alias for shutil.copy2
>
> I'll step through the code to explain the process.
>
>     for entry in restindex['file']:
>
> restindex['file'] should be a list of all your file keywords from the
> page being processed.
>
>
>     src = join(self.dir, dirname(filename), entry)
>
> ``join`` is an alias for ``os.path.join``.
>
> ``filename`` is the name of the file being processed. When we are
> processing an index file, it could actually be a path relative to
> ``self.dir`` if the ``index-file`` option is set.
>
> You were getting lots of  ``File "..." referenced by file keyword
> missing.`` errors - so this logic seems to be faulty ?
>
> If you add the print statement I suggest, then you should be able to
> tell what is happening. I'm sure the solution is simple enough - but I
> can't see it.
>
> Thanks
>
> Fuzzyman
> http://www.voidspace.org.uk/python/index.shtml

Re: [Rest2web-develop] RE: File Bug

[Michael F. <fuz...@gm...>]

On 06/04/06, Andrew Ittner <and...@us...> wrote:
>
> I love one-line fixes.
>
> Changed
>     src =3D join(self.dir, filename, entry)



But in SVN I had already changed that line to :

    src =3D join(self.dir, dirname(filename), entry)

??? Because I'd like to move the two chunks of code into a single function,
that dirname will be needed for the case of index files.

I'll make that change and you can test again. :-)

Fuzzyman
http://ww.voidspace.org.uk/python/index.shtml


to
>     src =3D join(self.dir, entry)
>
> It appears that the filename value set the path to
> "index.txt/thefilename.ext" instead of just "thefilename.ext".  Once I
> removed it, both the index and non-index pages copied files fine (so I di=
d
> not need to change the other loop's src line).
>
> Andrew
>
> > -----Original Message-----
> > From: Fuzzyman [mailto:fuz...@vo...]
> > Sent: Tuesday, April 04, 2006 2:34 AM
> > To: res...@li...; Andrew Ittner
> > Subject: Re: File Bug
> >
> >
> > Hello Andrew,
> >
> > I would certainly appreciate your eyes in tracking down this bug. I
> > don't *think* it can be platform specific though, as there is no
> > platform specific code (I use platform independent functions from
> > ``os.path`` and ``shutil`` to do all the work).
> >
> > I can see where the bug is happening - for you it seems to be where the
> > location of the source file is determined. I've pasted the full code in
> > below - and explained the logic of the first part which determines the
> > source location. You could add in a print statement to see where it
> > thinks the source file *should be*, which will give us a clue as to wha=
t
> > is happening.
> >
> > The code exists twice (which means it ought to be turned into a method
> > of Processor). Once in the ``process`` method and once in the
> > ``processindex`` method.
> >
> > When rest2web starts, it changes the current directory to be the 'start=
'
> > directory (top level of the site directory tree).
> >
> > As it goes through the files, the current directory being worked on is
> > stored in ``self.dir`` (self being the processor instance). At the top
> > level directory the following is set in ``__init__``, ``self.dir =3D
> > os.curdir``. (Is it possible that this is the cause of the problem ?)
> >
> > Following code taken from -
> > http://rest2web.python-hosting.com/file/trunk/rest2web/restprocessor.py
> > - from the process method.
> >
> > # also copy files (from restindex) into same dir as target
> > for entry in restindex['file']:
> >     src =3D join(self.dir, dirname(filename), entry)
> >     ## XXX Add a ``print src`` here :-)
> >     if not os.path.isfile(src):
> >         print ('File "%s" referenced by file keyword missing.' %
> >             entry)
> >         continue
> >     #
> >     target_file_name =3D url2pathname(target)
> >     targetfile =3D os.path.normpath(join(
> >         self.target, self.dir, target_file_name))
> >     dest =3D join(dirname(targetfile), split(entry)[1])
> >     comparefile =3D os.path.normpath(join(
> >         self.compare, self.dir, target_file_name))
> >     cmp =3D join(dirname(comparefile), split(entry)[1])
> >     if comparefiles(src, cmp):
> >         print 'File "%s" identical, skipping.' % entry
> >         continue
> >     if not isdir(dirname(dest)):
> >         os.makedirs(dirname(dest))
> >     print 'Copying "%s".' % entry
> >     copy2(src, dest)   # alias for shutil.copy2
> >
> > I'll step through the code to explain the process.
> >
> >     for entry in restindex['file']:
> >
> > restindex['file'] should be a list of all your file keywords from the
> > page being processed.
> >
> >
> >     src =3D join(self.dir, dirname(filename), entry)
> >
> > ``join`` is an alias for ``os.path.join``.
> >
> > ``filename`` is the name of the file being processed. When we are
> > processing an index file, it could actually be a path relative to
> > ``self.dir`` if the ``index-file`` option is set.
> >
> > You were getting lots of  ``File "..." referenced by file keyword
> > missing.`` errors - so this logic seems to be faulty ?
> >
> > If you add the print statement I suggest, then you should be able to
> > tell what is happening. I'm sure the solution is simple enough - but I
> > can't see it.
> >
> > Thanks
> >
> > Fuzzyman
> > http://www.voidspace.org.uk/python/index.shtml
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting
> language
> that extends applications into web and mobile media. Attend the live
> webcast
> and join the prime developer group breaking into this new coding
> territory!
> http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D110944&bid=3D241720&dat=
=3D121642
> _______________________________________________
> Rest2web-develop mailing list
> Res...@li...
> https://lists.sourceforge.net/lists/listinfo/rest2web-develop
>



--
http://www.Voidspace.org.uk
The Place where headspace meets cyberspace. Online resource site - covering
science, technology, computing, cyberpunk, psychology, spirituality, fictio=
n
and more.

---
http://www.Voidspace.org.uk/python/index.shtml
Python utilities, modules and apps.
Including Nanagram, Dirwatcher and more.
---
http://www.fuchsiashockz.co.uk
http://groups.yahoo.com/group/void-shockz
---

Everyone has talent. What is rare is the courage to follow talent
to the dark place where it leads. -Erica Jong
Ambition is a poor excuse for not having sense enough to be lazy.
         -Milan Kundera

RE: [Rest2web-develop] RE: File Bug

[Andrew I. <and...@us...>]

Here is some debug output for the root's index:
self.dir=; filename=index.txt; entry=rest2web140x62.gif;
src=index.txt/rest2web140x62.gif

And for projects/rest2web's index:
self.dir=projects/rest2web; filename=projects/rest2web/index.txt;
entry=halloffamelogo.png;
src=projects/rest2web/projects/rest2web/index.txt/halloffamelogo.png

This line:
src = join(self.dir, dirname(filename), entry)

will concatenate the current directory, AND the current directory of a file
in the current directory, and THEN the filename.  So you end up with
DIR/SUBDIR/DIR/SUBDIR/filename.ext.

Since you want to merge these 2 loops, perhaps I should wait to add a fix.
Here's what I recommend: compare self.dir and dirname(filename)'s results:
if they match, use only one.  If they don't, use one.  I think we need a
complete test suite on this one, because you talked about setting the
index-file option, which I do not use in combination with the file keyword.

Andrew

-----Original Message-----
From: res...@li...
[mailto:res...@li...]On Behalf Of Michael
Foord
Sent: Thursday, April 06, 2006 12:15 AM
To: res...@li...
Subject: Re: [Rest2web-develop] RE: File Bug





On 06/04/06, Andrew Ittner <and...@us...> wrote:
I love one-line fixes.

Changed
    src = join(self.dir, filename, entry)


But in SVN I had already changed that line to :

    src = join(self.dir, dirname(filename), entry)

??? Because I'd like to move the two chunks of code into a single function,
that dirname will be needed for the case of index files.

I'll make that change and you can test again. :-)

Fuzzyman
http://ww.voidspace.org.uk/python/index.shtml



to
    src = join(self.dir, entry)

It appears that the filename value set the path to
"index.txt /thefilename.ext" instead of just "thefilename.ext".  Once I
removed it, both the index and non-index pages copied files fine (so I did
not need to change the other loop's src line).

Andrew

> -----Original Message-----
> From: Fuzzyman [mailto:fuz...@vo...]
> Sent: Tuesday, April 04, 2006 2:34 AM
> To: res...@li...; Andrew Ittner
> Subject: Re: File Bug
>
>
> Hello Andrew,
>
> I would certainly appreciate your eyes in tracking down this bug. I
> don't *think* it can be platform specific though, as there is no
> platform specific code (I use platform independent functions from
> ``os.path`` and ``shutil`` to do all the work).
>
> I can see where the bug is happening - for you it seems to be where the
> location of the source file is determined. I've pasted the full code in
> below - and explained the logic of the first part which determines the
> source location. You could add in a print statement to see where it
> thinks the source file *should be*, which will give us a clue as to what
> is happening.
>
> The code exists twice (which means it ought to be turned into a method
> of Processor). Once in the ``process`` method and once in the
> ``processindex`` method.
>
> When rest2web starts, it changes the current directory to be the 'start'
> directory (top level of the site directory tree).
>
> As it goes through the files, the current directory being worked on is
> stored in ``self.dir`` (self being the processor instance). At the top
> level directory the following is set in ``__init__``, ``self.dir =
> os.curdir``. (Is it possible that this is the cause of the problem ?)
>
> Following code taken from -
> http://rest2web.python-hosting.com/file/trunk/rest2web/restprocessor.py
> - from the process method.
>
> # also copy files (from restindex) into same dir as target
> for entry in restindex['file']:
>     src = join(self.dir, dirname(filename), entry)
>     ## XXX Add a ``print src`` here :-)
>     if not os.path.isfile(src):
>         print ('File "%s" referenced by file keyword missing.' %
>             entry)
>         continue
>     #
>     target_file_name = url2pathname(target)
>     targetfile = os.path.normpath(join(
>         self.target, self.dir, target_file_name))
>     dest = join(dirname(targetfile), split(entry)[1])
>     comparefile = os.path.normpath(join(
>         self.compare, self.dir, target_file_name))
>     cmp = join(dirname(comparefile), split(entry)[1])
>     if comparefiles(src, cmp):
>         print 'File "%s" identical, skipping.' % entry
>         continue
>     if not isdir(dirname(dest)):
>         os.makedirs(dirname(dest))
>     print 'Copying "%s".' % entry
>     copy2(src, dest)   # alias for shutil.copy2
>
> I'll step through the code to explain the process.
>
>     for entry in restindex['file']:
>
> restindex['file'] should be a list of all your file keywords from the
> page being processed.
>
>
>     src = join(self.dir, dirname(filename), entry)
>
> ``join`` is an alias for ``os.path.join``.
>
> ``filename`` is the name of the file being processed. When we are
> processing an index file, it could actually be a path relative to
> ``self.dir`` if the ``index-file`` option is set.
>
> You were getting lots of  ``File "..." referenced by file keyword
> missing.`` errors - so this logic seems to be faulty ?
>
> If you add the print statement I suggest, then you should be able to
> tell what is happening. I'm sure the solution is simple enough - but I
> can't see it.
>
> Thanks
>
> Fuzzyman
> http://www.voidspace.org.uk/python/index.shtml

Re: [Rest2web-develop] RE: File Bug

[Fuzzyman <fuz...@vo...>]

Andrew Ittner wrote:
> Here is some debug output for the root's index:
> self.dir=; filename=index.txt; entry=rest2web140x62.gif;
> src=index.txt/rest2web140x62.gif
>
>   
So this works fine.

> And for projects/rest2web's index:
> self.dir=projects/rest2web; filename=projects/rest2web/index.txt;
> entry=halloffamelogo.png;
> src=projects/rest2web/projects/rest2web/index.txt/halloffamelogo.png
>
> This line:
> src = join(self.dir, dirname(filename), entry)
>
> will concatenate the current directory, AND the current directory of a file
> in the current directory, and THEN the filename.  So you end up with
> DIR/SUBDIR/DIR/SUBDIR/filename.ext.
>
>   
Yup - I understand.

I think the method ``get_real_restindex`` is doing this to the filename 
- so I will work out a generic method of solving this. I wonder why this 
worked for me though ?

> Since you want to merge these 2 loops, perhaps I should wait to add a fix.
> Here's what I recommend: compare self.dir and dirname(filename)'s results:
> if they match, use only one.  If they don't, use one.  I think we need a
> complete test suite on this one, because you talked about setting the
> index-file option, which I do not use in combination with the file keyword.
>   
I am not currently using the file keyword at all, I do however use the 
index-file keyword quite a lot, so it is important to me that this works 
correctly.

I will sort it out.

Thanks for your help in diagnosing this.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

> Andrew
>
> -----Original Message-----
> From: res...@li...
> [mailto:res...@li...]On Behalf Of Michael
> Foord
> Sent: Thursday, April 06, 2006 12:15 AM
> To: res...@li...
> Subject: Re: [Rest2web-develop] RE: File Bug
>
>
>
>
>
> On 06/04/06, Andrew Ittner <and...@us...> wrote:
> I love one-line fixes.
>
> Changed
>     src = join(self.dir, filename, entry)
>
>
> But in SVN I had already changed that line to :
>
>     src = join(self.dir, dirname(filename), entry)
>
> ??? Because I'd like to move the two chunks of code into a single function,
> that dirname will be needed for the case of index files.
>
> I'll make that change and you can test again. :-)
>
> Fuzzyman
> http://ww.voidspace.org.uk/python/index.shtml
>
>
>
> to
>     src = join(self.dir, entry)
>
> It appears that the filename value set the path to
> "index.txt /thefilename.ext" instead of just "thefilename.ext".  Once I
> removed it, both the index and non-index pages copied files fine (so I did
> not need to change the other loop's src line).
>
> Andrew
>
>   
>> -----Original Message-----
>> From: Fuzzyman [mailto:fuz...@vo...]
>> Sent: Tuesday, April 04, 2006 2:34 AM
>> To: res...@li...; Andrew Ittner
>> Subject: Re: File Bug
>>
>>
>> Hello Andrew,
>>
>> I would certainly appreciate your eyes in tracking down this bug. I
>> don't *think* it can be platform specific though, as there is no
>> platform specific code (I use platform independent functions from
>> ``os.path`` and ``shutil`` to do all the work).
>>
>> I can see where the bug is happening - for you it seems to be where the
>> location of the source file is determined. I've pasted the full code in
>> below - and explained the logic of the first part which determines the
>> source location. You could add in a print statement to see where it
>> thinks the source file *should be*, which will give us a clue as to what
>> is happening.
>>
>> The code exists twice (which means it ought to be turned into a method
>> of Processor). Once in the ``process`` method and once in the
>> ``processindex`` method.
>>
>> When rest2web starts, it changes the current directory to be the 'start'
>> directory (top level of the site directory tree).
>>
>> As it goes through the files, the current directory being worked on is
>> stored in ``self.dir`` (self being the processor instance). At the top
>> level directory the following is set in ``__init__``, ``self.dir =
>> os.curdir``. (Is it possible that this is the cause of the problem ?)
>>
>> Following code taken from -
>> http://rest2web.python-hosting.com/file/trunk/rest2web/restprocessor.py
>> - from the process method.
>>
>> # also copy files (from restindex) into same dir as target
>> for entry in restindex['file']:
>>     src = join(self.dir, dirname(filename), entry)
>>     ## XXX Add a ``print src`` here :-)
>>     if not os.path.isfile(src):
>>         print ('File "%s" referenced by file keyword missing.' %
>>             entry)
>>         continue
>>     #
>>     target_file_name = url2pathname(target)
>>     targetfile = os.path.normpath(join(
>>         self.target, self.dir, target_file_name))
>>     dest = join(dirname(targetfile), split(entry)[1])
>>     comparefile = os.path.normpath(join(
>>         self.compare, self.dir, target_file_name))
>>     cmp = join(dirname(comparefile), split(entry)[1])
>>     if comparefiles(src, cmp):
>>         print 'File "%s" identical, skipping.' % entry
>>         continue
>>     if not isdir(dirname(dest)):
>>         os.makedirs(dirname(dest))
>>     print 'Copying "%s".' % entry
>>     copy2(src, dest)   # alias for shutil.copy2
>>
>> I'll step through the code to explain the process.
>>
>>     for entry in restindex['file']:
>>
>> restindex['file'] should be a list of all your file keywords from the
>> page being processed.
>>
>>
>>     src = join(self.dir, dirname(filename), entry)
>>
>> ``join`` is an alias for ``os.path.join``.
>>
>> ``filename`` is the name of the file being processed. When we are
>> processing an index file, it could actually be a path relative to
>> ``self.dir`` if the ``index-file`` option is set.
>>
>> You were getting lots of  ``File "..." referenced by file keyword
>> missing.`` errors - so this logic seems to be faulty ?
>>
>> If you add the print statement I suggest, then you should be able to
>> tell what is happening. I'm sure the solution is simple enough - but I
>> can't see it.
>>
>> Thanks
>>
>> Fuzzyman
>> http://www.voidspace.org.uk/python/index.shtml
>>     
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> Rest2web-develop mailing list
> Res...@li...
> https://lists.sourceforge.net/lists/listinfo/rest2web-develop
>
>