Re: [Htmlparser-user] [operations with source code of a web page]

[Derrick O. <Der...@Ro...>]

01) Looking at the source code, the SiteCapturer code goes through a 
NodeIterator, but the Parser.parse (NodeFilter) method with a null 
filter would do the same thing.
            // fetch the page and gather the list of nodes
            mParser.setURL (url);
            try
            {
                list = new NodeList ();
                for (NodeIterator e = mParser.elements (); 
e.hasMoreNodes (); )
                    list.add (e.nextNode ()); // URL conversion occurs 
in the tags
            }
            catch (EncodingChangeException ece)
            {
                // fix bug #998195 SiteCatpurer just crashed
                // try again with the encoding now set correctly
                // hopefully mPages, mImages, mCopied and mFinished 
won't be corrupted
                mParser.reset ();
                list = new NodeList ();
                for (NodeIterator e = mParser.elements (); 
e.hasMoreNodes (); )
                    list.add (e.nextNode ());
            }

02) No validation is done on the page. However, the heuristics built in 
to the tag parsing will insert terminating nodes (identified by 0 == 
(tag.getEndPosition () - tag.getStartPosition ())) where end tags are 
required.

03) After parsing the entire page, the source is available (as 
characters characters or String) from the Page/Source, which is exposed 
on the parser as getPage(). Strings in Java are UTF-16 encoded unicode. 
Any errors in conversion (using the encoding specified by the HTTP 
header or HTML meta tags) will already have been committed by then.


myer wrote:

>Hello dear users and developers,
>
>  currently I write my bachelor's thesis, where I use the
>  functionality of HTML Parser. In my program I need almost the same
>  result as SiteCapturer does. So I've started to learn how it works
>  and change it for my project. But some moments are not fully clear
>  to me.
>
>  01) How does HTML Parser obtain source code of a web page before
>  parsing? In the following I will speak about the SiteCapturer
>  example. Does it start with a 'null' filter to get all the nodes of a
>  web page for the very first time? And only then applies other
>  filters indicated by user. Or it parses 'on the fly': gets the first
>  node of a source, compares with node filter, and only if it
>  successfully passes the filter check saves it into a data structure.
>  Say, node list. What I need to do, is to get the whole 'untouched'
>  source code of a web page before parsing. Should I go the way
>  mentioned in this thread
>    http://sourceforge.net/forum/message.php?msg_id=3005740
>  or there are any other more intelligent solutions? Perhaps there
>  exist any already implemented method? Something like
>  page.getSource()? How does the SiteCapturer solve this problem?
>
>  02) Is the source code of a web page normalized anyhow before the
>  actual parsing? Are there any attempts made to supply a parser with
>  a validated HTML source? Or is it better to use products of other
>  developers, e.g. JTidy?
>
>  03) Also I would like to save the source code of a web page in its
>  original encoding or in Unicode. I do not want to lose any
>  international character of the source. I need to save the source of a
>  page into the database and be able to obtain it in its original form
>  if necessary. Does HTML Parser supports source code convertion into
>  Unicode?
>
>  
>