#2 Returned HTML is not correct

[Anonymous]

I use curl to login in to a Merchant's Account and Scrap certain data from that account. I follow the site's login procedures using cURL. The problem arises when I accessing one of the site's pages. In some cases, the page's html returned by cURL is correct ( Note: I use the html to parse certain data from it ) and in some cases it does not and i get a server error returned from the page. Why does that happen ? Waiting for replies. and below is the code I use to get that page:

CurlGlue cg = new CurlGlue();

cg.setopt(CURL.OPT_URL, step.getUrl());

// Set type of user agent the site will handle with
cg.setopt(CURL.OPT_USERAGENT, merchantModel.getAgent());

// Set number of fields that will be posted to url
cg.setopt(CURL.OPT_POST, parameters.size());

// Set curl to establish a fresh connection for this step
cg.setopt(CURL.OPT_FRESH_CONNECT, 1);
// Forbid reuse of this connection
cg.setopt(CURL.OPT_FORBID_REUSE, 1);

cg.setopt(CURL.OPT_POSTFIELDS, postFields);

//Set curl to allow redirection if performed by url
cg.setopt(CURL.OPT_FOLLOWLOCATION, 1);

cg.setopt(CURL.OPT_CAINFO,
merchantModel.getCertificateUrl());

cg.setopt(CURL.OPT_COOKIEJAR,
merchantModel.getCookieUrl());
cg.setopt(CURL.OPT_COOKIEFILE,
merchantModel.getCookieUrl());

cg.setopt(CURL.OPT_WRITEFUNCTION, this);
cg.perform();

// After that I use parse the content of the stringbuffer that was filled by the handleString method from the CurlWrite interface