Post problems

[Anonymous]

Hello

Like some others I have some problems with the http post method...

I have identified the different parameters to post but it still not work.

Could it be possible that the destination page is looking for a specific name
of the form, and in that case how could I define it ?

Thank's

[Steven P. Goldsmith]

This is tricky since some sites use a different a action url once you visit or
login to a page or it's look for a certain User-Agent in the http header. I
can save you come time in this respect (note I'm using 2.1 trunk, so the xml
will look different). Consider the following code fragment which will visit a
page to extract action URL and other post parameters (also look for my cookie
fix for the 2.1 trunk code a few posts down) in order to post required
parameters:

<config xmlns="[url]http://web-harvest.sourceforge.net/schema/2.1/core[/url]" charset="UTF-8">

    <def var="userAgent">Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)</def>
    <def var="xmlDoc"/>
    <def var="postUrl"/>
    <def var="bowStEvent"/>

    <script>
        <![CDATA[
        // Date format
        java.text.SimpleDateFormat simpleDateFormat = new java.text.SimpleDateFormat("MM/dd/yyyy");
        // Get current Date as String
        String dateStr = simpleDateFormat.format(new Date());        
        ]]>
    </script>

     <!-- Parse common page elements into variables defined above -->

    <function name="parseCommon">    
        <!-- Get POST URL from form's action attribute -->
        <set var="postUrl">
            <xpath expression="//form/@action">
                <value-of expr="${xmlDoc}"/>  
            </xpath>
        </set>
        <!-- Get _bowStEvent from input value attribute -->
        <set var="bowStEvent">
            <xpath expression="//input[@name='_bowStEvent']/@value">
                <value-of expr="${xmlDoc}"/>  
            </xpath>
        </set>    
    </function>

    <!-- Visit page, so we can extract common post elements -->

    <set var="xmlDoc">
        <html-to-xml>
            <http method="get" url="[url]http://www.manateesheriff.com/wps/portal/PublicInterest/ActiveCalls[/url]" follow-redirects="true">
                <http-header name="User-Agent">
                    <value-of expr="${userAgent}"/>
                </http-header>
            </http>
        </html-to-xml>
    </set>

    <!-- Parse common page elements into variables -->

    <call name="parseCommon"/>

    <!-- Get a list of TR elements of name DataContainer using URL postUrl -->

    <set var="xmlDoc">
        <xpath expression='//tr[@name="DataContainer"]'>
            <html-to-xml>
                <http method="post" url="www.manateesheriff.com${postUrl}" follow-redirects="true">
                    <http-header name="User-Agent">
                        <value-of expr="${userAgent}"/>  
                    </http-header>
                    <http-param name="_bowStEvent">
                        <value-of expr="${bowStEvent}"/>  
                    </http-param>
                    <http-param name="Street"></http-param>
                    <http-param name="DateFrom">
                        <value-of expr="${dateStr}"/>  
                    </http-param>
                    <http-param name="DateTo">
                        <value-of expr="${dateStr}"/>  
                    </http-param>
                    <http-param name="submit_button">Submit</http-param>
                </http>
            </html-to-xml>
        </xpath>
    </set>

    <file action="write" path="/home/someuser/mcso_active_calls_output.xml">
        <value-of expr="${xmlDoc}"/>
    </file>

</config>

[Anonymous]

Thank you for your code, I'll analyze it and I'll try to do something similar.

You are using a parameter: follow-redirects

What is it for ?

[Anonymous]

An important detail ... Where (how) do I get the 2.1 trunk version ?

[Steven P. Goldsmith]

follow-redirects does what it says, follow http redirects the site may
process. This is defaulted to true in 2b1, but is false as default in 2.1
trunk. https://web-harvest.svn.sourceforge.net/svnroot/web-
harvest/trunk

When you compile Maven project you will notice saxon/saxon-dom are not in
Maven repos. You need to add to your local Maven or Nexus repository then it
will build without error.

[Anonymous]

Compiling the project is a new world to me... sorry.

Is there any specific o general help that explain how to do it (get the local
repository and compile) ?

Tank's

[Steven P. Goldsmith]

Just scan the forums or google Maven build. Just install Maven as directed,
put mvn in execution path, cd to where pom.xml file run mvn with appropriate
target.

[Alex Wajda]

Guys, the build should be fixed now :)

Compiling the project is a new world to me... sorry.

Eric, you would be surprised how easy that world in WH 2.1 :)
As Steven said in the previous post - just install Maven and then execute "mvn
install" command from the WH project folder (where pom.xml is located).

[Anonymous]

I go into it ...

Thank's for your explanations.