[MinGW-notify] [ mingw-Bugs-3432808 ] mingw-get.exe: unexpected status: code = 304

[SF/projects/mingw n. l. <min...@li...>]

Bugs item #3432808, was opened at 2011-11-03 06:44
Message generated for change (Comment added) made by keithmarshall
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=102435&aid=3432808&group_id=2435

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: mingw-get
Group: Waiting User Response
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Robert Hartmann (ifrh)
Assigned to: Keith Marshall (keithmarshall)
Summary: mingw-get.exe:  unexpected status: code = 304 

Initial Comment:
Hi there,

behind a proxy (squid/2.6.STABLE5) I tried yesterday and today (3.Nov.2011)
to install MinGW with mingw-get-inst-20110802.exe on Windows XP prof SP3 as Administrator into a new not existent directory (c:\MinGW).

I choose the option "Use pre-packaged repository catalogues  20110802"

I selected the components C Compiler, C++ Compiler, Fortran Compiler and ObjCC Compiler.


From the mingw-get process I got warnings and error messages while downloading this two files:
gcc-4.5.2-1-mingw32-lic.tar.lzma
gcc-fortran-4.5.2-1-mingw32-bin.tar.lzma

After ending of the installation process these files are missing in C:\MinGW\var\cache\mingw-get\packages .

The concrete messages are:

************************
gcc-4.5.2-1-mingw32-lic.tar.lzma:
************************
mingw-get.exe: *** WARNING *** http://prdownloads.sourceforge.net/mingw/gcc-4.5.2-1-mingw32-lic.tar.lzma?download: opened with unexpected status: code = 304
mingw-get.exe: *** WARNING *** please report this to the mingw-get maintainer
mingw-get.exe: *** ERROR *** Get package: http://prdownloads.sourceforge.net/mingw/gcc-4.5.2-1-mingw32-lic.tar.lzma?download: download failed
************************

************************
gcc-fortran-4.5.2-1-mingw32-bin.tar.lzma
************************
mingw-get.exe: *** WARNING *** http://prdownloads.sourceforge.net/mingw/gcc-fortran-4.5.2-1-mingw32-bin.tar.lzma?download: opened with unexpected status: code = 304
mingw-get.exe: *** WARNING *** please report this to the mingw-get maintainer
mingw-get.exe: *** ERROR *** Get package: http://prdownloads.sourceforge.net/mingw/gcc-fortran-4.5.2-1-mingw32-bin.tar.lzma?download: download failed
************************


I think there shouldn't be a "If-Modified-Since" in HTTP-Header
at the GET command in the first mingw-get run, because in
C:\MinGW\var\cache\mingw-get\packages
there are no files at first.


In the attached files you can see

mingw-get-inst.txt: mingw-get's messages (copy & paste)

lic1.txt: wireshark's protocoll of
GET http://dfn.dl.sourceforge.net/project/mingw/MinGW/BaseSystem/GCC/Version4/gcc-4.5.2-1/gcc-4.5.2-1-mingw32-lic.tar.lzma HTTP/1.0\r\n

lic1a.txt: wireshark's protocoll of the response.

Best regards,
Robert Hartmann (Germany)

----------------------------------------------------------------------

>Comment By: Keith Marshall (keithmarshall)
Date: 2011-11-21 14:39

Message:
Hi Robert,

> But I wasn't able to run configure.ac.

configure.ac isn't runnable; it's the input DATA file for autoconf.  Since
configure itself isn't stored in CVS, you need to run autoconf, in the top
source directory, to generate it, so yes, you'll also need to mingw-get
install mingw32-autoconf if you don't already have it.

> I tried it with
> [1] bash from DJGPP
> [2] sh from UnxUtils

You likely won't have much luck with either of these.  Use this...

> [3] bash from msys

...or a mingw32 cross-compiler under Cygwin, Linux or *BSD.

Once you've created the configure script, simply by running:

  $ autoconf

then you may:

  $ mkdir obj
  $ cd obj
  $ ../configure DEBUGLEVEL=0x0fff CFLAGS='-g -O0 -mms-bitfields'
  $ make

(adding suitable --build and --host options to configure, if cross
compiling).

----------------------------------------------------------------------

Comment By: Robert Hartmann (ifrh)
Date: 2011-11-21 11:22

Message:
Hi Keith,
You asked "So, I guess libz-dev and liblzma-dev installed okay?" Yes, they
were ok.
And now I could download mingw32-bzip2-dev 
Thanks.

But I wasn't able to run configure.ac. 

I tried it with 
[1] bash from DJGPP 
[2] sh from UnxUtils
[3] bash from msys
all gave me "./configure.ac: line 25: syntax error near unexpected token
..."
for complete  messages please see in attached file "configure-problem.txt"

I am not familiar with the autoconfig tools nor with bash-script-language.

Have I got something wrong?

Best regards, Robert

----------------------------------------------------------------------

Comment By: Keith Marshall (keithmarshall)
Date: 2011-11-20 11:24

Message:
Hi Robert,

So, I guess libz-dev and liblzma-dev installed okay?

> mingw-get install mingw32-libbz2-dev
> mingw-get: *** ERROR *** mingw32-libbz2-dev: unknown package
>
> Has the package an other name?

Sorry, I've misled you.  While the DLL package is indeed libbz2-dll,
anomalously the corresponding development kit is furnished by bzip2-dev. 
It should work better with:

mingw-get install mingw32-bzip2-dev

(or even just):

mingw-get install bzip2-dev

----------------------------------------------------------------------

Comment By: Robert Hartmann (ifrh)
Date: 2011-11-20 09:58

Message:
Hi Keith,
You told me to download: "mingw32-libz-dev, mingw32-libbz2-dev,and
mingw32-liblzma-dev" I tried it just now from home. Because I am ill at the
moment, I can't test, that means I can not concentrate on something.
But downloading files shouldn't be to hard for me:

mingw-get install mingw32-libbz2-dev
mingw-get: *** ERROR *** mingw32-libbz2-dev: unknown package

Has the package an other name? 

Which files from 
https://sourceforge.net/projects/mingw/files/MinGW/Extension/bzip2/bzip2-1.0.6-4/
should I download manualy 

Or did you mean, that I had to get the libraries from CVS like mingw-get?

Best regards, Robert

----------------------------------------------------------------------

Comment By: Keith Marshall (keithmarshall)
Date: 2011-11-10 14:04

Message:
You should follow the instructions here:
https://sourceforge.net/scm/?type=cvs&group_id=2435

to perform an anonymous checkout of the mingw-get module; (you may use the
MSYS cvs client).

You'll also need to install mingw32-libz-dev, mingw32-libbz2-dev,and
mingw32-liblzma-dev, after which you use the standard 'configure && make'
process to build it -- that automatically takes care of the library and
include flags.

FTR, I configure in a obj/ subdir of the top level source directory, with
flags:

  ../configure CFLAGS='-g -O0 -mms-bitflags' DEBUGLEVEL=0x0fff

for a debug build, and then switch to:

  ../configure CFLAGS='-s -O2 -mms-bitflags' LDFLAGS=-s DEBUGLEVEL=0x0fff

for release; (I also add '--build=linux --host=mingw32', because I'm
cross-compiling -- you don't need that when building with MSYS, in its
MinGW personality).

----------------------------------------------------------------------

Comment By: Robert Hartmann (ifrh)
Date: 2011-11-10 12:05

Message:
Hi Keith Marshall,
yes, I am willing to build from source. And I like to assist with
debugging. 

Would you please give me some days ? And a big pointer to what libraries I
should link. -lwininet and what others. I should download the complete
folder  http://mingw.cvs.sourceforge.net/viewvc/mingw/mingw-get/src/ and
its subdirectories  and what else I need to compile?

----------------------------------------------------------------------

Comment By: Keith Marshall (keithmarshall)
Date: 2011-11-08 03:43

Message:
Hi Robert,

Thanks for the protocol analysis.  Are you willing to build from source, to
assist with debugging?  Will the attached patch suffice?

----------------------------------------------------------------------

Comment By: Robert Hartmann (ifrh)
Date: 2011-11-07 07:00

Message:
Hi Keith Marshall,

I searched around in the internet and found that Internet Explorer on
requests to a special URL by the user usually contacts the web server for
updates to that page via a conditional If-Modified-Since request, if the
page is still inside the disk cache ("Temporary Internet Files")  [1]

I assume this is also true for wininet.dll which mingw-get is using. 

I verified, that the If-Modified-Since was inserted to the HTTP-Header by
wininet.dll:
First I cleaned up (but forgot to backup) the IE-cache.
Than I started Wireshark and ran mingw-get-inst.exe for installation into a
fresh directory.
Wireshark DOESN'T protocol any „If-Modified-Since“-requests nor
„Not-Modified“-responses.
Next step was deleting the MinGW installation but not touching the
IE-cache.
Now I run mingw-get-inst again and Wireshark DOES protocol
„If-Modified-Since“-requests for 
every file which was in IE-cache. 

By the way in section 13.5.2 Non-modifiable Headers [2] is written: 
A transparent proxy SHOULD NOT modify an end-to-end header unless the
definition of that header requires or specifically allows that. 
A transparent proxy MUST NOT modify any of the following fields in a
request or response, and it MUST NOT add any of these fields if not already
present: 
      - Content-Location
      - Content-MD5
      - ETag
      - Last-Modified
A transparent proxy MUST NOT modify any of the following fields in a
response: 
      - Expires
but it MAY add any of these fields if not already present.

That means, the If-Modified-Since header in request from my IP with
http.user_agent containing "MinGW Installer" comes really from
mingw-get.exe.

Now, why is wininet.dll allowed to send a „If-Modified-Since“? You
pointed me to the codefile.

The message, which occures comes from codeline 484 in pkginet.cpp in
function pkgInternetAgent::OpenURL.

There at some places ResourceStatus is tested against HTTP_STATUS_OK.

In line 399 it is defined ResourceStatus = QueryStatus( ResourceHandle );
The function QueryStatus is calling HttpQueryInfo with
HTTP_QUERY_STATUS_CODE.

„HTTP_QUERY_STATUS_CODE Receives the status code returned by the
server.“ [see 3]

That means you must be prepared to not only receive failure-codes >=400 or
the Success-Code  HTTP_STATUS_OK but also the server-status code
Not-Modified (304). 
You are handling 304 „Not-Modified“ like a failure code but it is not. 

From a server's point of view  „304“ is an  OK–statuscode. 

It seems to me from analysing the wireshark protocol that wininet.dll is
following redirection response-codes 
HTTP_STATUS_MOVED (301), HTTP_STATUS_REDIRECT(302),
HTTP_STATUS_REDIRECT_METHOD(303), HTTP_STATUS_USE_PROXY(305),
HTTP_STATUS_REDIRECT_KEEP_VERB(307) automaticaly without your intervention;
i.e. the GET-Command to http://prdownloads.sourceforge.net switched to
http://downloads.sourceforge.net and than moved to
http://dfn.dl.sourceforge.net

I really think that you should handle HTTP_STATUS_NOT_MODIFIED (304; The
requested resource has not been modified) the same way as  HTTP_STATUS_OK
(200; the request completed successfully). That means download the file :)


Or you can handle 304 in a different way than 200, like in this simplified
pseudocode algorithm:

%%%%%%%%%%%
response=OpenUrl_And_CheckStatus(URL);
filename=TrimFilenameFromURL(URL);
swich(response){
case 200: DownloadFile_and-save-to_Mingw-cache(URL); break;
case 304: GetFileFromIEcache_and-save-to_Mingw-cache(filename);
                if Not_in(filename,mingw-get-cache) {
                      RemoveFilenameFromIEcache(filename);
                      DownloadFile_and-save-to_Mingw-cache(URL);
                 } break;
default handle_other_HTTP-codes(response);
}

if Not_in(filename,mingw-get-cache)  report error message
%%%%%%%%%%%

For implementing  GetFileFromIEcache_and-save-to_Mingw-cache and 
RemoveFilenameFromIEcache it is perhapse useful to access the
INTERNET_CACHE_ENTRY_INFO structure [5].
Useful functions are FindFirstUrlCacheEntry function [6], 
DeleteUrlCacheEntry [7] and GetUrlCacheEntryInfo function [8].

Best regards,
Robert Hartmann

[1] http://support.microsoft.com/kb/234067/en-us
[2] http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html
[3] http://msdn.microsoft.com/en-us/library/ms906347.aspx 
[4] http://msdn.microsoft.com/en-us/site/aa384325
[5] http://msdn.microsoft.com/en-us/library/aa385134%28v=VS.85%29.aspx
[6] http://msdn.microsoft.com/en-us/library/aa384026%28v=VS.85%29.aspx
[7] http://msdn.microsoft.com/en-us/library/aa383983%28v=VS.85%29.aspx
[8] http://msdn.microsoft.com/en-us/library/aa384185%28v=VS.85%29.aspx

----------------------------------------------------------------------

Comment By: Keith Marshall (keithmarshall)
Date: 2011-11-05 11:10

Message:
> I think there shouldn't be a "If-Modified-Since" in HTTP-Header
> at the GET command in the first mingw-get run,

mingw-get is not the progenitor of that header.  I'm by no means an expert
on proxy server protocols, but as I understand it, typically, this header
would be inserted by your squid proxy, when it already holds a cached copy
of the requested content, and is checking for a possible upstream update
before simply delivering its cached copy.

A further, perhaps less likely possibility, is that this header is being
inserted by internet explorer, (mingw-get uses Microsoft's wininet.dll
library, which is a component of the internet explorer protocol stack); in
this case, we would expect the requested content to be delivered from
internet explorer's local cache.

> because in
> C:\MinGW\var\cache\mingw-get\packages
> there are no files at first.

mingw-get's package cache is completely independent of both your squid
proxy cache, and internet explorer's download's cache; it's where mingw-get
will ultimately store the downloaded package archives, regardless of
whether they are delivered directly from the internet, or from a copy
previously cached by internet explorer or by a proxy server.

You can find all of mingw-get's download specific code in the
src/pkginet.cpp module, within the source package or CVS.  I can see
nothing in that which would explicitly insert an If-Modified-Since header. 
Unless you can show me otherwise, I'm assuming that the header has been
added by either internet explorer, or by your squid proxy server.  In
either case, the agent inserting the header is responsible for delivering
the requested content from its own cache, if it chooses not to download
afresh.  I really don't see how this is a mingw-get bug, or what action
mingw-get might take to circumvent it.

----------------------------------------------------------------------

Comment By: Robert Hartmann (ifrh)
Date: 2011-11-04 01:05

Message:
Hi, for completeness, I uploade additionally the Wireshark protocol for the
HTTP request 
and the response of GET command for
gcc-fortran-4.5.2-1-mingw32-bin.tar.lzma.

If you look into the request (fort2.txt) you can see:
If-Modified-Since: Wed, 12 Jan 2011 21:50:25 GMT; length=5397546

If you look into the response (fort2a.txt) you can see:
Last-Modified: Wed, 12 Jan 2011 21:44:33 GMT

I think because of the wrong Date and/or Time values in If-Modified-Since
neighter 
gcc-4.5.2-1-mingw32-lic.tar.lzma nor
gcc-fortran-4.5.2-1-mingw32-bin.tar.lzma
can be downloaded by mingw-get.exe with the "pre-packaged repository
catalogues 20110802"
from mingw-get-inst-20110802.exe.

Best regards, 
Robert Hartmann 


----------------------------------------------------------------------

Comment By: Robert Hartmann (ifrh)
Date: 2011-11-03 12:47

Message:
Hi,
If you hava a look into the attached file lic1.txt than you can read:

If-Modified-Since: Sat, 22 Jan 2011 22:54:22 GMT; length=93652\r\n

Just now I looked at the Date information for
gcc-4.5.2-1-mingw32-lic.tar.lzma  at 
http://sourceforge.net/projects/mingw/files/MinGW/BaseSystem/GCC/Version4/gcc-4.5.2-1/
Here you can see  Date  2011-01-12  and size 21.2 kB.
Because 2011-01-12 is NOT later than 22 Jan 2011 the file wasn't
downloaded.

Please correct the Date from If-Modified-Since  in HTTP-Header of
GET-command  or remove the If-Modified-Since.

Tomorrow I can have a look to the Date in the If-Modiefied-Since from the
GET command for gcc-fortran-4.5.2-1-mingw32-bin.tar.lzma. I think it is
wrong too.

Best regards, Robert

----------------------------------------------------------------------

Comment By: Robert Hartmann (ifrh)
Date: 2011-11-03 06:48

Message:
Perhaps this bug has todo with bug 3413249 -- but I don't think so...

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=102435&aid=3432808&group_id=2435