GetGnuWin32 no longer works

trekker99
2008-09-20
2013-05-28
1 2 > >> (Page 1 of 2)
  • trekker99
    trekker99
    2008-09-20

    Hi,

    With the makeover of Sourceforge, the package no longer works. Any idea how to patch download.bat to get it to work again?

    Thanks,

    Edward

     
    • Rob Davenport
      Rob Davenport
      2008-10-03

      Yes, the makeover really screwed GetGnuWin32 up.   All the URLs are not in the project list page, but buried one more level. 

      The first problem with download.bat is in line 331 where the author correct states that it is likely to break with changed HTML.  I fiddled with the sed script and could get output:

      <code>
      bin\sed -n "/^<td\s\+class\s*=\s*""""download""""/{n;s/.*/ &/p};/^<a href=""""http:\/project\/showfiles.*"""">$/Is//\1/p" normal.html >getgnuwin32.tmp
      </code>

      but that's only the start of the problem.   The output is just URLs to /project/showfiles.php links.  Download.bat will have to put in another pass to download each of THOSE pages and then scan THEM for the actual downloads.sourceforge.net links (they are there, but it'll be a little trickier to parse them out), and THEN it can build it's sorted list and run through it to download the files.    Certainly not impossible, but more work than I have time for at the moment.  Maybe this clue will help someone else get it to work.

      Rob

       
    • ferret
      ferret
      2008-10-05

      I have cobbled together a script - very alpha code - but it seems to work for doing the download part.

      ------ SNIP script starts --------
      rem New GetGnuWin32 Download Script -
      rem 5 Oct 2008

      for /F "usebackq tokens=1-4* delims=/ " %%a in (`Echo %date%`) do SET DateStr=%%b-%%c-%%d
      for /F "usebackq tokens=1-4* delims=:." %%a in (`Echo %time%`) do SET TimeStr=%%a-%%b_%%c

      rem do not use quotes around file paths - these are added in the script
      rem Main Path of GetGnuWin32 Updater Script
      Set MainPath=%CD%
      rem Path where old Packages are moved to
      Set OldPkg=%MainPath%\oldpacks
      Rem Path where New Packages are located
      Set NewPkg=%MainPath%\packages
      Rem Path where tools are located used by this script
      Set binGnu=%MainPath%\bin
      Rem Path where temp file are located
      Set TmpGnu=%MainPath%\tmp
      Rem Path where HTML Files from Sourceforge are downloaded to
      Set TmpHtml=%TmpGnu%\HTML
      Rem Path to config Files
      Set confGnu=%MainPath%\conf
      Rem Path and filename for Logfile used with wget and other logging
      Set LogGnu=%TmpGnu%\LogFile.txt
      Rem File used for storing list of current packages we have.
      Set CurPkgList=%TmpGnu%\CurPkgsLst.txt
      Rem File used to list packages we want to exclude from download
      Set ExcPkgList=exclude.txt
      Rem File used to list packages we want to include from previous downloads
      Set IncPkgList=include.txt

      Rem These are Various Flags we can set

      Rem Delete Old Pkgs rather than save in a Date stamped folder
      Rem if Set to a non blank value the old Packages are deleted rather than kept.
      rem Set DelOldPkgs=Y
      Set DelOldPkgs=

      Rem Configs for Wget
      Rem current options set are create a logfile and append to it, enable use of proxy.
      Set WgetOpts=--append-output="%LogGnu%" --proxy
      Rem Set your proxy to use with Wget
      Set http_proxy=http://10.0.0.12:8080/

      rem Download Main SourceForge GnuWin32 Download Page URL
      Set GnuWin32Page=http://sourceforge.net/project/showfiles.php?group_id=23617

      Rem Build MissingFolders
      IF NOT Exist "%OldPkg%" md "%OldPkg%"
      Rem Path where New Packages are located
      IF NOT Exist "%NewPkg%" MD "%NewPkg%" 
      Rem Path where temp file are located
      IF NOT Exist "%TmpGnu%" md "%TmpGnu%"
      Rem Path where HTML Files from Sourceforge are downloaded to
      IF NOT Exist "%TmpHtml%" md "%TmpHtml%"
      Rem Path to config Files
      IF NOT Exist "%confGnu%" md "%confGnu%"

      Rem Clean TmpFolders for new run
      IF EXIST "%TmpGnu%\*.*" del "%TmpGnu%\*.*" /Q
      IF Exist "%TmpHtml%\*.*" del "%TmpHtml%\*.*" /Q

      Rem Grab the List of all the files in the %NewPkg% location and Store in %CurPkgList% for later use to remove any packages
      Rem that haven't changed, we won't redownload them instead we will move the from the oldPkg location.
      Echo Building Current Local Package List....
      Echo %date%,%time%,Building Current Local Package List from %NewPkg% into %CurPkgList%>> %LogGnu%
      dir "%NewPkg%" /B /ON > "%CurPkgList%"

      rem Move the Old Packages from %OldPkg%
      IF "%DelOldPkgs%"=="" Echo Deleting old Pacakges from %OldPkg%
      IF "%DelOldPkgs%"=="" Echo %date%,%time%,Deleting old Pacakges from %OldPkg%>> %LogGnu% 
      IF "%DelOldPkgs%"=="" del "%OldPkg%\*.*" /Q
      IF "%DelOldPkgs%"=="" Goto :OvrMvOldPkg
      IF NOT EXIST "%OldPkg%\*.*" Goto :MvNewPkg
      Echo Moving Old Packages from %OldPkg% to %OldPkg%\%DateStr%
      Echo %date%,%time%,Moving Old Packages from %OldPkg% to %OldPkg%\%DateStr%>> %LogGnu%
      Rem Here we create a new folder in oldPkg and move the files into it
      Rem then we move the files in %NewPkg% to %OldPkg%
      MD "%OldPkg%\%DateStr%"
      Attrib -R "%OldPkg%\*.*"
      Move "%OldPkg%\*.*" "%OldPkg%\%DateStr%" > nul
      :MvNewPkg
      IF NOT EXIST "%NewPkg%\*.*" Goto :OvrMvOldPkg
      Echo Moving Current Packages from %NewPkg% to Old Package  Dir %OldPkg%
      Echo %date%,%time%,Moving Current Packages from %NewPkg% to Old Package  Dir %OldPkg%>> %LogGnu%
      Attrib -R "%NewPkg%\*.*"
      Move "%NewPkg%\*.*" "%OldPkg%">nul
      :OvrMvOldPkg

      :DownloadMainSrcDlPage
      Rem Download Main Project Download Page
      Echo %date%,%time%,Starting Download of Main Project Page:%GnuWin32Page%>> %LogGnu%
      Echo Starting Download of Main Project Page:%GnuWin32Page%

      rem Download the Sourceforge Page to download.html
      "%binGnu%\wget.exe" %WgetOpts% --output-document="%TmpGnu%\download.html" "%GnuWin32Page%"

      Echo %date%,%time%,Processing downloaded file looking for HTML File paths....>> %LogGnu%
      Echo Processing downloaded file looking for HTML File paths....
      Rem now we strip it down to the source HTML Pages we need
      rem ==========================================================================================================
      rem this block gives me file name in format arc-nnnn
      "%binGnu%\sed.exe" -n "/showfiles.php/p"  "%TmpGnu%\download.html" > "%TmpGnu%\ListOfSrc.html"
      "%binGnu%\sed.exe" -n "/td/p" "%TmpGnu%\ListOfSrc.html" >  "%TmpGnu%\ListOfSrc.2.html" 
      "%binGnu%\sed.exe" -r -e "s/\x22\x3e/\x22\x3e\n/g" "%TmpGnu%\ListOfSrc.2.html" > "%TmpGnu%\ListOfSrc.3.html"
      "%binGnu%\sed.exe" -r -n "/\x2fa/p" "%TmpGnu%\ListOfSrc.3.html" >  "%TmpGnu%\ListOfSrc.4.html"
      "%binGnu%\sed.exe"  -e "s/<\x2fa><\x2ftd>//g"  "%TmpGnu%\ListOfSrc.4.html" >  "%TmpGnu%\ListOfSrc.5.html"
      "%binGnu%\sed.exe" -n "/Download/!p"   "%TmpGnu%\ListOfSrc.5.html" >  "%TmpGnu%\ListOfSrc.6.html"
      rem  rem Join two lines together
      "%binGnu%\sed.exe"  -e "$!N;s/\n/-/"   "%TmpGnu%\ListOfSrc.6.html" >  "%TmpGnu%\ListOfSrc.7.html"
      rem at end of filename add .html"
      "%binGnu%\sed.exe"  -e "s/$/\.html\x22/" "%TmpGnu%\ListOfSrc.7.html" >  "%TmpGnu%\ListOfSrc.8.html"
      copy /b "%TmpGnu%\ListOfSrc.8.html" "%TmpGnu%\filenames.txt" > nul
      rem ==========================================================================================================
      rem Generate Local Filenames for moving current packages into %NewPkg%
      "%binGnu%\sed.exe" -e "s/\.html\x22//"  "%TmpGnu%\filenames.txt" > "%TmpGnu%\PkgNames.txt"

      rem ==========================================================================================================
      rem ==========================================================================================================
      rem rem Looking to get next URL for list of Filenames for Dl.
      rem rem  "sed.exe" -n "/showfiles.php/p"  "%TmpGnu%\download.html" > "%TmpGnu%\ListOfSrc.html"
      rem rem  "sed.exe" -n "/td/p" "%TmpGnu%\ListOfSrc.html" >  "%TmpGnu%\ListOfSrc.2.html" 
      rem rem "sed.exe" -r -e "s/\x22\x3e/\x22\x3e\n/g" "%TmpGnu%\ListOfSrc.2.html" > "%TmpGnu%\ListOfSrc.3.html"
      "%binGnu%\sed.exe" -n "/release_id/p" "%TmpGnu%\ListOfSrc.3.html" >  "%TmpGnu%\ListOfSrc.9.html"
      rem rem delete <a href="
      "%binGnu%\sed.exe" -r -e "s/\x3ca href\x3d\x22//g" "%TmpGnu%\ListOfSrc.9.html" >  "%TmpGnu%\ListOfSrc.10.html"
      rem rem delete ">
      "%binGnu%\sed.exe" -r -e "s/\x22\x3e/\x22/g" "%TmpGnu%\ListOfSrc.10.html" >  "%TmpGnu%\ListOfSrc.11.html"
      rem rem remove lines with <td>
      "%binGnu%\sed.exe" -r -n "/\x3ctd\x3e/!p" "%TmpGnu%\ListOfSrc.11.html" >  "%TmpGnu%\ListOfSrc.12.html"
      rem rem Add the line http://sourceforge.net/ to the start
      "%binGnu%\sed.exe" -r -e "s@^@\x22http://sourceforge.net@" "%TmpGnu%\ListOfSrc.12.html" >  "%TmpGnu%\ListOfSrc.13.html"
      rem rem remove the ^&amp; lines
      "%binGnu%\sed.exe" -r -e "s/amp;//g" "%TmpGnu%\ListOfSrc.13.html" >  "%TmpGnu%\ListOfSrc.14.html"
      copy /b "%TmpGnu%\ListOfSrc.14.html" "%TmpGnu%\URL-List.txt" > nul
      rem ==========================================================================================================
      rem Now Combine the 2 together
      rem rem Add the lines of each file together
      "%binGnu%\paste" "%TmpGnu%\filenames.txt" "%TmpGnu%\URL-List.txt" > "%TmpGnu%\Get-Urls.txt

      Rem Need to setup SedTmpHtml this is double slashed pathing for passing through Sed
      Rem and Also the BinPath for Wget, this enables us to pass it through the next SED lines
      Echo %TmpHtml% | "%binGnu%\sed.exe" -e "s/\\/\\\\/g;s/^/@Set sedTmpHtml=/;s/[ \t]*$//"> "%TmpGnu%\SetSedTmpHtml.cmd"
      Echo %binGnu% | "%binGnu%\sed.exe" -e "s/\\/\\\\/g;s/^/@Set sedbinGnu=/;s/[ \t]*$//">> "%TmpGnu%\SetSedTmpHtml.cmd"
      call "%TmpGnu%\SetSedTmpHtml.cmd"
      del "%TmpGnu%\SetSedTmpHtml.cmd"

      rem rem Now Added the wget command to the file
      "%binGnu%\sed.exe" -r -e "s/^/\x22%sedbinGnu%\\wget.exe\x22 \x25WgetOpts\x25 --output-document=\x22%sedTmpHtml%\\/" "%TmpGnu%\Get-Urls.txt" > "%TmpGnu%\Wget-HTML.1.cmd"

      Rem Process Exclusion List
      Rem This removes all project files we do not want to download.
      IF NOT EXIST "%confGnu%\%ExcPkgList%" Goto :OvrExcludeList
      rem Exclusion List exists lets process it.
      rem First Remove lines with a Hash at the start
      "%binGnu%\sed.exe" -n "/^#/!p" "%confGnu%\%ExcPkgList%" > "%TmpGnu%\exclude.1.txt"
      rem now remove the trailing # and following text
      "%binGnu%\sed.exe" -e "s/#.*$//" "%TmpGnu%\exclude.1.txt" > "%TmpGnu%\exclude.2.txt"
      rem Remove Leading/trailing whitespace from the lines that are left also remove empty lines. 
      "%binGnu%\sed.exe" -r -e "s/^[ \t]*//;s/[ \t]*$//;/^$/d" "%TmpGnu%\exclude.2.txt" > "%TmpGnu%\exclude.3.txt"
      copy /b "%TmpGnu%\exclude.3.txt" "%TmpGnu%\%ExcPkgList%" > nul
      rem Now Cleanup the URLS List from the %ExcPkgList% remove from Wget-HTML.cmd
      copy /b "%TmpGnu%\Wget-HTML.1.cmd" "%TmpGnu%\Wget-HTML.2.cmd" > nul
      for /F "usebackq" %%a IN ("%TmpGnu%\%ExcPkgList%") DO "%binGnu%\sed.exe" -i -n "/\\%%a-/!p" "%TmpGnu%\Wget-HTML.2.cmd"
      copy /b "%TmpGnu%\Wget-HTML.2.cmd" "%TmpGnu%\Wget-HTML.cmd" > nul
      :OvrExcludeList

      Rem Move Current Packages we already have from %OldPkg% to %NewPkg%
      Echo %date%,%time%,Moving Current Packages from %OldPkg% to %NewPkg%....>> %LogGnu%
      Echo Moving Current Packages from %OldPkg% to %NewPkg%....
      For /F "usebackq" %%a in ( "%TmpGnu%\PkgNames.txt" ) DO IF EXIST "%OldPkg%\%%a*.*" move "%OldPkg%\%%a*.*" "%NewPkg%" > nul 
      rem Now list the files we already have, we will use this later to remove the URLS we don;t need to download.
      dir "%NewPkg%" /B /ON > "%TmpGnu%\PkgsAlreadyGot.txt"

      rem Process the includes list move the files into the %NewPkg% folder.
      IF NOT EXIST "%confGnu%\%IncPkgList%" Goto :OvrIncludeList
      rem Inclusion List exists lets process it.
      rem First Remove lines with a Hash at the start
      "%binGnu%\sed.exe" -n "/^#/!p" "%confGnu%\%IncPkgList%" > "%TmpGnu%\include.1.txt"
      rem now remove the trailing # and following text
      "%binGnu%\sed.exe" -e "s/#.*$//" "%TmpGnu%\include.1.txt" > "%TmpGnu%\include.2.txt"
      rem Remove Leading/trailing whitespace from the lines that are left.
      "%binGnu%\sed.exe" -r -e "s/^[ \t]*//;s/[ \t]*$//" "%TmpGnu%\include.2.txt" > "%TmpGnu%\include.3.txt"
      copy /b "%TmpGnu%\include.3.txt" "%TmpGnu%\%IncPkgList%" > nul
      For /F "usebackq" %%a in ( "%TmpGnu%\%IncPkgList%" ) DO IF EXIST "%OldPkg%\%%a.*" move "%OldPkg%\%%a.*" "%NewPkg%"
      :OvrIncludeList

      Rem
      Rem Now Download all the Pages where the download links are for the actual files Download URLS Are located
      Rem
      Echo Please wait downloading the HTML Files with the URLS to the binaries....
      Echo %date%,%Time%,Starting Download of HTML Files with Download files list...>> "%LogGnu%"
      Echo on
      @call "%TmpGnu%\Wget-HTML.cmd">> "%LogGnu%"
      @Echo off
      Echo %date%,%Time%,Finished Download of HTML Files with Download files list...>> "%LogGnu%"

      Rem
      Rem Now we need to combine then extract the file list ready for download.
      Rem
      Echo %date%,%time%,Building Download List....>> "%LogGnu%"
      Echo Building Download List....

      If exist "%TmpGnu%\MasterDlList.1.txt" del "%TmpGnu%\MasterDlList.1.txt"
      For %%a in ("%TmpHtml%\*.*") do Type %%a >> "%TmpGnu%\MasterDlList.1.txt"
      rem strip down to only sourceforce downloads.sourceforge.net
      "%binGnu%\sed.exe" -n "/downloads.sourceforge.net/p"  "%TmpGnu%\MasterDlList.1.txt" > "%TmpGnu%\MasterDlList.2.txt"
      rem Strip out URLS
      rem find href="
      "%binGnu%\sed.exe" -e "s/href=/\n/g" "%TmpGnu%\MasterDlList.2.txt" > "%TmpGnu%\MasterDlList.3.txt"
      rem find  onClick
      "%binGnu%\sed.exe" -e "s/onClick/\n/g" "%TmpGnu%\MasterDlList.3.txt" > "%TmpGnu%\MasterDlList.4.txt"
      rem Keep Downloads Links
      "%binGnu%\sed.exe" -n "/downloads\./p" "%TmpGnu%\MasterDlList.4.txt" > "%TmpGnu%\MasterDlList.5.txt"
      rem Pull out (bin\|doc\|lib\|dep\)\.zip
      rem "%binGnu%\sed.exe" -n "/bin\|doc\|lib\|dep\.zip/p" "%TmpGnu%\MasterDlList.5.txt" > "%TmpGnu%\MasterDlList.6.txt"
      "%binGnu%\sed.exe" -r -n "/(-bin|-doc|-lib|-dep)\.zip/p" "%TmpGnu%\MasterDlList.5.txt" > "%TmpGnu%\MasterDlList.6.txt"
      rem remove exe files 
      "%binGnu%\sed.exe" -n "/.zip/p" "%TmpGnu%\MasterDlList.6.txt" > "%TmpGnu%\MasterDlList.7.txt"
      rem remove amp;
      "%binGnu%\sed.exe" -e "s/amp\;//g" "%TmpGnu%\MasterDlList.7.txt" > "%TmpGnu%\MasterDlList.8.txt"
      rem strip "" chars
      "%binGnu%\sed.exe" -e "s/\x22//g" "%TmpGnu%\MasterDlList.8.txt" > "%TmpGnu%\MasterDlList.9.txt"
      copy /b "%TmpGnu%\MasterDlList.9.txt" "%TmpGnu%\MasterDlList.txt"

      rem now get the files
      Echo %date%,%time%,Started Downloading the Files....>> "%LogGnu%"
      Echo Downloading the Files.... 
      rem No Directories

      "%binGnu%\wget.exe" %WgetOpts% -nd --directory-prefix="%NewPkg%" -c --input-file="%TmpGnu%\MasterDlList.txt"
      Echo %date%,%time%,Completed Downloading the Files....>> "%LogGnu%"
      Echo Downloading the Files....Finished. 
         
       

       
    • ferret
      ferret
      2008-10-09

      I have a patch to fix a couple of issues I've found with the script thats been thrown together

      use patch< download.patch to update

      =====Snip Start=====
      --- download-original.cmd    Sun Oct 05 20:42:18 2008
      +++ download-new.cmd    Wed Oct 08 18:27:51 2008
      @@ -1,5 +1,13 @@
      -rem New GetGnuWin32 Download Script - Ferret@techie.com
      -rem 5 Oct 2008
      +@echo off
      +rem New GetGnuWin32 Download Script
      +rem 5 Oct 2008
      +
      +rem
      +rem requires from WinGnuWin32
      +rem   sed.exe V:GNU sed version 4.1.5
      +rem   paste.exe V:paste (GNU coreutils) 5.3.0
      +rem   wget.exe V:GNU Wget 1.10.1
      +rem

      for /F "usebackq tokens=1-4* delims=/ " %%a in (`Echo %date%`) do SET DateStr=%%b-%%c-%%d
      for /F "usebackq tokens=1-4* delims=:." %%a in (`Echo %time%`) do SET TimeStr=%%a-%%b_%%c
      @@ -20,7 +28,7 @@
      Rem Path to config Files
      Set confGnu=%MainPath%\conf
      Rem Path and filename for Logfile used with wget and other logging
      -Set LogGnu=%TmpGnu%\LogFile.txt
      +Set LogGnu=%TmpGnu%\LogFile-%DateStr%.txt
      Rem File used for storing list of current packages we have.
      Set CurPkgList=%TmpGnu%\CurPkgsLst.txt
      Rem File used to list packages we want to exclude from download
      @@ -33,11 +41,12 @@
      Rem Delete Old Pkgs rather than save in a Date stamped folder
      Rem if Set to a non blank value the old Packages are deleted rather than kept.
      rem Set DelOldPkgs=Y
      -Set DelOldPkgs=
      +Set DelOldPkgs=N

      Rem Configs for Wget
      Rem current options set are create a logfile and append to it, enable use of proxy.
      -Set WgetOpts=--append-output="%LogGnu%" --proxy
      +Set WgetOpts=--append-output="%LogGnu%" --no-proxy
      +rem  --proxy
      Rem Set your proxy to use with Wget
      Set http_proxy=http://10.0.0.12:8080/

      @@ -45,6 +54,15 @@
      Set GnuWin32Page=http://sourceforge.net/project/showfiles.php?group_id=23617

      +rem Check for File Dependencies.
      +Set FileMissing=N
      +For %%a in (sed.exe paste.exe wget.exe ) Do IF NOT EXIST "%binGnu%\%%a" Set FileMissing=Y
      +IF "%FileMissing%"=="N" Goto :DependsOk
      +Echo Files Required are missing! Cannot Continue.
      +echo One of the following files is missing:  sed.exe paste.exe wget.exe From %binGnu%
      +pause
      +Goto :End
      +:DependsOk
      Rem Build MissingFolders
      IF NOT Exist "%OldPkg%" md "%OldPkg%"
      Rem Path where New Packages are located
      @@ -63,39 +81,46 @@
      Rem Grab the List of all the files in the %NewPkg% location and Store in %CurPkgList% for later use to remove any packages
      Rem that haven't changed, we won't redownload them instead we will move the from the oldPkg location.
      Echo Building Current Local Package List....
      -Echo %date%,%time%,Building Current Local Package List from %NewPkg% into %CurPkgList%>> %LogGnu%
      +Echo %date%,%time%,Building Current Local Package List from %NewPkg% into %CurPkgList%>> "%LogGnu%""
      dir "%NewPkg%" /B /ON > "%CurPkgList%"

      rem Move the Old Packages from %OldPkg%
      -IF "%DelOldPkgs%"=="" Echo Deleting old Pacakges from %OldPkg%
      -IF "%DelOldPkgs%"=="" Echo %date%,%time%,Deleting old Pacakges from %OldPkg%>> %LogGnu% 
      -IF "%DelOldPkgs%"=="" del "%OldPkg%\*.*" /Q
      -IF "%DelOldPkgs%"=="" Goto :OvrMvOldPkg
      -IF NOT EXIST "%OldPkg%\*.*" Goto :MvNewPkg
      +set FndOldPkgs=N
      +for %%a in ("%OldPkg%\*.*")do set FndOldPkgs=Y
      +IF "%FndOldPkgs%"=="N" Goto :MvNewPkg
      +IF "%DelOldPkgs%"=="Y" Echo Deleting old Pacakges from %OldPkg%
      +IF "%DelOldPkgs%"=="Y" Echo %date%,%time%,Deleting old Pacakges from %OldPkg%>> "%LogGnu%"
      +IF "%DelOldPkgs%"=="Y" del "%OldPkg%\*.*" /Q
      +IF "%DelOldPkgs%"=="Y" Goto ::MvNewPkg
      +
      Echo Moving Old Packages from %OldPkg% to %OldPkg%\%DateStr%
      -Echo %date%,%time%,Moving Old Packages from %OldPkg% to %OldPkg%\%DateStr%>> %LogGnu%
      +Echo %date%,%time%,Moving Old Packages from %OldPkg% to %OldPkg%\%DateStr%>> "%LogGnu%"
      Rem Here we create a new folder in oldPkg and move the files into it
      Rem then we move the files in %NewPkg% to %OldPkg%
      MD "%OldPkg%\%DateStr%"
      Attrib -R "%OldPkg%\*.*"
      -Move "%OldPkg%\*.*" "%OldPkg%\%DateStr%" > nul
      +Move "%OldPkg%\*.*" "%OldPkg%\%DateStr%" > "%LogGnu%"
      :MvNewPkg
      -IF NOT EXIST "%NewPkg%\*.*" Goto :OvrMvOldPkg
      +set FndNewPkgs=N
      +for %%a in ("%NewPkg%\*.*")do set FndNewPkgs=Y
      +IF "%FndNewPkgs%"=="N" Goto :OvrMvNewPkg
      Echo Moving Current Packages from %NewPkg% to Old Package  Dir %OldPkg%
      -Echo %date%,%time%,Moving Current Packages from %NewPkg% to Old Package  Dir %OldPkg%>> %LogGnu%
      +Echo %date%,%time%,Moving Current Packages from %NewPkg% to Old Package  Dir %OldPkg%>> "%LogGnu%"
      Attrib -R "%NewPkg%\*.*"
      -Move "%NewPkg%\*.*" "%OldPkg%">nul
      -:OvrMvOldPkg
      +Move "%NewPkg%\*.*" "%OldPkg%">>"%LogGnu%"
      +:OvrMvNewPkg
      +set FndOldPkgs=
      +set FndNewPkgs=

      :DownloadMainSrcDlPage
      Rem Download Main Project Download Page
      -Echo %date%,%time%,Starting Download of Main Project Page:%GnuWin32Page%>> %LogGnu%
      +Echo %date%,%time%,Starting Download of Main Project Page:%GnuWin32Page%>> "%LogGnu%""
      Echo Starting Download of Main Project Page:%GnuWin32Page%

      rem Download the Sourceforge Page to download.html
      "%binGnu%\wget.exe" %WgetOpts% --output-document="%TmpGnu%\download.html" "%GnuWin32Page%"

      -Echo %date%,%time%,Processing downloaded file looking for HTML File paths....>> %LogGnu%
      +Echo %date%,%time%,Processing downloaded file looking for HTML File paths....>> "%LogGnu%"
      Echo Processing downloaded file looking for HTML File paths....
      Rem now we strip it down to the source HTML Pages we need
        rem ==========================================================================================================
      @@ -167,7 +192,7 @@
      :OvrExcludeList

        Rem Move Current Packages we already have from %OldPkg% to %NewPkg%
      -Echo %date%,%time%,Moving Current Packages from %OldPkg% to %NewPkg%....>> %LogGnu%
      +Echo %date%,%time%,Moving Current Packages from %OldPkg% to %NewPkg%....>> "%LogGnu%"
      Echo Moving Current Packages from %OldPkg% to %NewPkg%....
      For /F "usebackq" %%a in ( "%TmpGnu%\PkgNames.txt" ) DO IF EXIST "%OldPkg%\%%a*.*" move "%OldPkg%\%%a*.*" "%NewPkg%" > nul 
        rem Now list the files we already have, we will use this later to remove the URLS we don;t need to download.
      @@ -191,8 +216,11 @@
      Rem
      Echo Please wait downloading the HTML Files with the URLS to the binaries....
      Echo %date%,%Time%,Starting Download of HTML Files with Download files list...>> "%LogGnu%"
      +
      Echo on
      -@call "%TmpGnu%\Wget-HTML.cmd">> "%LogGnu%"
      +@call "%TmpGnu%\Wget-HTML.cmd"
      +
      +rem >> "%LogGnu%.2"
      @Echo off
      Echo %date%,%Time%,Finished Download of HTML Files with Download files list...>> "%LogGnu%"

      @@ -205,7 +233,7 @@
      Echo Building Download List....

      If exist "%TmpGnu%\MasterDlList.1.txt" del "%TmpGnu%\MasterDlList.1.txt"
      -For %%a in ("%TmpHtml%\*.*") do Type %%a >> "%TmpGnu%\MasterDlList.1.txt"
      +For %%a in ("%TmpHtml%\*.*") do Type "%%a" >> "%TmpGnu%\MasterDlList.1.txt"
        rem strip down to only sourceforce downloads.sourceforge.net
      "%binGnu%\sed.exe" -n "/downloads.sourceforge.net/p"  "%TmpGnu%\MasterDlList.1.txt" > "%TmpGnu%\MasterDlList.2.txt"
        rem Strip out URLS
      @@ -224,17 +252,20 @@
      "%binGnu%\sed.exe" -e "s/amp\;//g" "%TmpGnu%\MasterDlList.7.txt" > "%TmpGnu%\MasterDlList.8.txt"
        rem strip "" chars
      "%binGnu%\sed.exe" -e "s/\x22//g" "%TmpGnu%\MasterDlList.8.txt" > "%TmpGnu%\MasterDlList.9.txt"
      -copy /b "%TmpGnu%\MasterDlList.9.txt" "%TmpGnu%\MasterDlList.txt"
      +Rem Remove file that we already have from Download list.
      +copy /b "%TmpGnu%\MasterDlList.9.txt" "%TmpGnu%\MasterDlList.10.txt"
      +for /F "usebackq" %%a IN ("%TmpGnu%\PkgsAlreadyGot.txt") DO "%binGnu%\sed.exe" -i -n "/%%a/!p" "%TmpGnu%\MasterDlList.10.txt"
      +copy /b "%TmpGnu%\MasterDlList.10.txt" "%TmpGnu%\MasterDlList.txt"
       
        rem now get the files
      Echo %date%,%time%,Started Downloading the Files....>> "%LogGnu%"
      Echo Downloading the Files.... 
        rem No Directories
      -
      +rem pause
      "%binGnu%\wget.exe" %WgetOpts% -nd --directory-prefix="%NewPkg%" -c --input-file="%TmpGnu%\MasterDlList.txt"
      Echo %date%,%time%,Completed Downloading the Files....>> "%LogGnu%"
      Echo Downloading the Files....Finished. 
          
        

      -
      +:End

      =====Snip End=====

       
      • Dwaine Gonyier
        Dwaine Gonyier
        2008-10-14

        I tried to use the patch on the previously supplied download cmd file and I get the following error:

        E:\tmp\gnuwin32\GetGnuWin32>patch < download.patch
        patching file download-original.cmd
        patch: **** malformed patch at line 17: for /F "usebackq tokens=1-4* delims=/ "%%a in (`Echo %date%`) do SET DateStr=%%b-%%c-%%d

         
    • ferret
      ferret
      2008-10-16

      Oops here is the full script

      ================== Start Snip ==================
      @echo off
      rem New GetGnuWin32 Download Script - Ferret@techie.com
      rem 5 Oct 2008

      rem
      rem requires from WinGnuWin32
      rem   sed.exe V:GNU sed version 4.1.5
      rem   paste.exe V:paste (GNU coreutils) 5.3.0
      rem   wget.exe V:GNU Wget 1.10.1
      rem

      for /F "usebackq tokens=1-4* delims=/ " %%a in (`Echo %date%`) do SET DateStr=%%b-%%c-%%d
      for /F "usebackq tokens=1-4* delims=:." %%a in (`Echo %time%`) do SET TimeStr=%%a-%%b_%%c

      rem do not use quotes around file paths - these are added in the script
      rem Main Path of GetGnuWin32 Updater Script
      Set MainPath=%CD%
      rem Path where old Packages are moved to
      Set OldPkg=%MainPath%\oldpacks
      Rem Path where New Packages are located
      Set NewPkg=%MainPath%\packages
      Rem Path where tools are located used by this script
      Set binGnu=%MainPath%\bin
      Rem Path where temp file are located
      Set TmpGnu=%MainPath%\tmp
      Rem Path where HTML Files from Sourceforge are downloaded to
      Set TmpHtml=%TmpGnu%\HTML
      Rem Path to config Files
      Set confGnu=%MainPath%\conf
      Rem Path and filename for Logfile used with wget and other logging
      Set LogGnu=%TmpGnu%\LogFile-%DateStr%.txt
      Rem File used for storing list of current packages we have.
      Set CurPkgList=%TmpGnu%\CurPkgsLst.txt
      Rem File used to list packages we want to exclude from download
      Set ExcPkgList=exclude.txt
      Rem File used to list packages we want to include from previous downloads
      Set IncPkgList=include.txt

      Rem These are Various Flags we can set

      Rem Delete Old Pkgs rather than save in a Date stamped folder
      Rem if Set to a non blank value the old Packages are deleted rather than kept.
      rem Set DelOldPkgs=Y
      Set DelOldPkgs=N

      Rem Configs for Wget
      Rem current options set are create a logfile and append to it, enable use of proxy.
      Set WgetOpts=--append-output="%LogGnu%" --no-proxy
      rem  --proxy
      Rem Set your proxy to use with Wget
      Set http_proxy=http://10.0.0.12:8080/

      rem Download Main SourceForge GnuWin32 Download Page URL
      Set GnuWin32Page=http://sourceforge.net/project/showfiles.php?group_id=23617

      rem Check for File Dependencies.
      Set FileMissing=N
      For %%a in (sed.exe paste.exe wget.exe ) Do IF NOT EXIST "%binGnu%\%%a" Set FileMissing=Y
      IF "%FileMissing%"=="N" Goto :DependsOk
      Echo Files Required are missing! Cannot Continue.
      echo One of the following files is missing:  sed.exe paste.exe wget.exe From %binGnu%
      pause
      Goto :End
      :DependsOk
      Rem Build MissingFolders
      IF NOT Exist "%OldPkg%" md "%OldPkg%"
      Rem Path where New Packages are located
      IF NOT Exist "%NewPkg%" MD "%NewPkg%" 
      Rem Path where temp file are located
      IF NOT Exist "%TmpGnu%" md "%TmpGnu%"
      Rem Path where HTML Files from Sourceforge are downloaded to
      IF NOT Exist "%TmpHtml%" md "%TmpHtml%"
      Rem Path to config Files
      IF NOT Exist "%confGnu%" md "%confGnu%"

      Rem Clean TmpFolders for new run
      IF EXIST "%TmpGnu%\*.*" del "%TmpGnu%\*.*" /Q
      IF Exist "%TmpHtml%\*.*" del "%TmpHtml%\*.*" /Q

      Rem Grab the List of all the files in the %NewPkg% location and Store in %CurPkgList% for later use to remove any packages
      Rem that haven't changed, we won't redownload them instead we will move the from the oldPkg location.
      Echo Building Current Local Package List....
      Echo %date%,%time%,Building Current Local Package List from %NewPkg% into %CurPkgList%>> "%LogGnu%""
      dir "%NewPkg%" /B /ON > "%CurPkgList%"

      rem Move the Old Packages from %OldPkg%
      set FndOldPkgs=N
      for %%a in ("%OldPkg%\*.*")do set FndOldPkgs=Y
      IF "%FndOldPkgs%"=="N" Goto :MvNewPkg
      IF "%DelOldPkgs%"=="Y" Echo Deleting old Pacakges from %OldPkg%
      IF "%DelOldPkgs%"=="Y" Echo %date%,%time%,Deleting old Pacakges from %OldPkg%>> "%LogGnu%"
      IF "%DelOldPkgs%"=="Y" del "%OldPkg%\*.*" /Q
      IF "%DelOldPkgs%"=="Y" Goto ::MvNewPkg

      Echo Moving Old Packages from %OldPkg% to %OldPkg%\%DateStr%
      Echo %date%,%time%,Moving Old Packages from %OldPkg% to %OldPkg%\%DateStr%>> "%LogGnu%"
      Rem Here we create a new folder in oldPkg and move the files into it
      Rem then we move the files in %NewPkg% to %OldPkg%
      MD "%OldPkg%\%DateStr%"
      Attrib -R "%OldPkg%\*.*"
      Move "%OldPkg%\*.*" "%OldPkg%\%DateStr%" > "%LogGnu%"
      :MvNewPkg
      set FndNewPkgs=N
      for %%a in ("%NewPkg%\*.*")do set FndNewPkgs=Y
      IF "%FndNewPkgs%"=="N" Goto :OvrMvNewPkg
      Echo Moving Current Packages from %NewPkg% to Old Package  Dir %OldPkg%
      Echo %date%,%time%,Moving Current Packages from %NewPkg% to Old Package  Dir %OldPkg%>> "%LogGnu%"
      Attrib -R "%NewPkg%\*.*"
      Move "%NewPkg%\*.*" "%OldPkg%">>"%LogGnu%"
      :OvrMvNewPkg
      set FndOldPkgs=
      set FndNewPkgs=

      :DownloadMainSrcDlPage
      Rem Download Main Project Download Page
      Echo %date%,%time%,Starting Download of Main Project Page:%GnuWin32Page%>> "%LogGnu%""
      Echo Starting Download of Main Project Page:%GnuWin32Page%

      rem Download the Sourceforge Page to download.html
      "%binGnu%\wget.exe" %WgetOpts% --output-document="%TmpGnu%\download.html" "%GnuWin32Page%"

      Echo %date%,%time%,Processing downloaded file looking for HTML File paths....>> "%LogGnu%"
      Echo Processing downloaded file looking for HTML File paths....
      Rem now we strip it down to the source HTML Pages we need
      rem ==========================================================================================================
      rem this block gives me file name in format arc-nnnn
      "%binGnu%\sed.exe" -n "/showfiles.php/p"  "%TmpGnu%\download.html" > "%TmpGnu%\ListOfSrc.html"
      "%binGnu%\sed.exe" -n "/td/p" "%TmpGnu%\ListOfSrc.html" >  "%TmpGnu%\ListOfSrc.2.html" 
      "%binGnu%\sed.exe" -r -e "s/\x22\x3e/\x22\x3e\n/g" "%TmpGnu%\ListOfSrc.2.html" > "%TmpGnu%\ListOfSrc.3.html"
      "%binGnu%\sed.exe" -r -n "/\x2fa/p" "%TmpGnu%\ListOfSrc.3.html" >  "%TmpGnu%\ListOfSrc.4.html"
      "%binGnu%\sed.exe"  -e "s/<\x2fa><\x2ftd>//g"  "%TmpGnu%\ListOfSrc.4.html" >  "%TmpGnu%\ListOfSrc.5.html"
      "%binGnu%\sed.exe" -n "/Download/!p"   "%TmpGnu%\ListOfSrc.5.html" >  "%TmpGnu%\ListOfSrc.6.html"
      rem  rem Join two lines together
      "%binGnu%\sed.exe"  -e "$!N;s/\n/-/"   "%TmpGnu%\ListOfSrc.6.html" >  "%TmpGnu%\ListOfSrc.7.html"
      rem at end of filename add .html"
      "%binGnu%\sed.exe"  -e "s/$/\.html\x22/" "%TmpGnu%\ListOfSrc.7.html" >  "%TmpGnu%\ListOfSrc.8.html"
      copy /b "%TmpGnu%\ListOfSrc.8.html" "%TmpGnu%\filenames.txt" > nul
      rem ==========================================================================================================
      rem Generate Local Filenames for moving current packages into %NewPkg%
      "%binGnu%\sed.exe" -e "s/\.html\x22//"  "%TmpGnu%\filenames.txt" > "%TmpGnu%\PkgNames.txt"

      rem ==========================================================================================================
      rem ==========================================================================================================
      rem rem Looking to get next URL for list of Filenames for Dl.
      rem rem  "sed.exe" -n "/showfiles.php/p"  "%TmpGnu%\download.html" > "%TmpGnu%\ListOfSrc.html"
      rem rem  "sed.exe" -n "/td/p" "%TmpGnu%\ListOfSrc.html" >  "%TmpGnu%\ListOfSrc.2.html" 
      rem rem "sed.exe" -r -e "s/\x22\x3e/\x22\x3e\n/g" "%TmpGnu%\ListOfSrc.2.html" > "%TmpGnu%\ListOfSrc.3.html"
      "%binGnu%\sed.exe" -n "/release_id/p" "%TmpGnu%\ListOfSrc.3.html" >  "%TmpGnu%\ListOfSrc.9.html"
      rem rem delete <a href="
      "%binGnu%\sed.exe" -r -e "s/\x3ca href\x3d\x22//g" "%TmpGnu%\ListOfSrc.9.html" >  "%TmpGnu%\ListOfSrc.10.html"
      rem rem delete ">
      "%binGnu%\sed.exe" -r -e "s/\x22\x3e/\x22/g" "%TmpGnu%\ListOfSrc.10.html" >  "%TmpGnu%\ListOfSrc.11.html"
      rem rem remove lines with <td>
      "%binGnu%\sed.exe" -r -n "/\x3ctd\x3e/!p" "%TmpGnu%\ListOfSrc.11.html" >  "%TmpGnu%\ListOfSrc.12.html"
      rem rem Add the line http://sourceforge.net/ to the start
      "%binGnu%\sed.exe" -r -e "s@^@\x22http://sourceforge.net@" "%TmpGnu%\ListOfSrc.12.html" >  "%TmpGnu%\ListOfSrc.13.html"
      rem rem remove the ^&amp; lines
      "%binGnu%\sed.exe" -r -e "s/amp;//g" "%TmpGnu%\ListOfSrc.13.html" >  "%TmpGnu%\ListOfSrc.14.html"
      copy /b "%TmpGnu%\ListOfSrc.14.html" "%TmpGnu%\URL-List.txt" > nul
      rem ==========================================================================================================
      rem Now Combine the 2 together
      rem rem Add the lines of each file together
      "%binGnu%\paste" "%TmpGnu%\filenames.txt" "%TmpGnu%\URL-List.txt" > "%TmpGnu%\Get-Urls.txt

      Rem Need to setup SedTmpHtml this is double slashed pathing for passing through Sed
      Rem and Also the BinPath for Wget, this enables us to pass it through the next SED lines
      Echo %TmpHtml% | "%binGnu%\sed.exe" -e "s/\\/\\\\/g;s/^/@Set sedTmpHtml=/;s/[ \t]*$//"> "%TmpGnu%\SetSedTmpHtml.cmd"
      Echo %binGnu% | "%binGnu%\sed.exe" -e "s/\\/\\\\/g;s/^/@Set sedbinGnu=/;s/[ \t]*$//">> "%TmpGnu%\SetSedTmpHtml.cmd"
      call "%TmpGnu%\SetSedTmpHtml.cmd"
      del "%TmpGnu%\SetSedTmpHtml.cmd"

      rem rem Now Added the wget command to the file
      "%binGnu%\sed.exe" -r -e "s/^/\x22%sedbinGnu%\\wget.exe\x22 \x25WgetOpts\x25 --output-document=\x22%sedTmpHtml%\\/" "%TmpGnu%\Get-Urls.txt" > "%TmpGnu%\Wget-HTML.1.cmd"

      Rem Process Exclusion List
      Rem This removes all project files we do not want to download.
      IF NOT EXIST "%confGnu%\%ExcPkgList%" Goto :OvrExcludeList
      rem Exclusion List exists lets process it.
      rem First Remove lines with a Hash at the start
      "%binGnu%\sed.exe" -n "/^#/!p" "%confGnu%\%ExcPkgList%" > "%TmpGnu%\exclude.1.txt"
      rem now remove the trailing # and following text
      "%binGnu%\sed.exe" -e "s/#.*$//" "%TmpGnu%\exclude.1.txt" > "%TmpGnu%\exclude.2.txt"
      rem Remove Leading/trailing whitespace from the lines that are left also remove empty lines. 
      "%binGnu%\sed.exe" -r -e "s/^[ \t]*//;s/[ \t]*$//;/^$/d" "%TmpGnu%\exclude.2.txt" > "%TmpGnu%\exclude.3.txt"
      copy /b "%TmpGnu%\exclude.3.txt" "%TmpGnu%\%ExcPkgList%" > nul
      rem Now Cleanup the URLS List from the %ExcPkgList% remove from Wget-HTML.cmd
      copy /b "%TmpGnu%\Wget-HTML.1.cmd" "%TmpGnu%\Wget-HTML.2.cmd" > nul
      for /F "usebackq" %%a IN ("%TmpGnu%\%ExcPkgList%") DO "%binGnu%\sed.exe" -i -n "/\\%%a-/!p" "%TmpGnu%\Wget-HTML.2.cmd"
      copy /b "%TmpGnu%\Wget-HTML.2.cmd" "%TmpGnu%\Wget-HTML.cmd" > nul
      :OvrExcludeList

      Rem Move Current Packages we already have from %OldPkg% to %NewPkg%
      Echo %date%,%time%,Moving Current Packages from %OldPkg% to %NewPkg%....>> "%LogGnu%"
      Echo Moving Current Packages from %OldPkg% to %NewPkg%....
      For /F "usebackq" %%a in ( "%TmpGnu%\PkgNames.txt" ) DO IF EXIST "%OldPkg%\%%a*.*" move "%OldPkg%\%%a*.*" "%NewPkg%" > nul 
      rem Now list the files we already have, we will use this later to remove the URLS we don;t need to download.
      dir "%NewPkg%" /B /ON > "%TmpGnu%\PkgsAlreadyGot.txt"

      rem Process the includes list move the files into the %NewPkg% folder.
      IF NOT EXIST "%confGnu%\%IncPkgList%" Goto :OvrIncludeList
      rem Inclusion List exists lets process it.
      rem First Remove lines with a Hash at the start
      "%binGnu%\sed.exe" -n "/^#/!p" "%confGnu%\%IncPkgList%" > "%TmpGnu%\include.1.txt"
      rem now remove the trailing # and following text
      "%binGnu%\sed.exe" -e "s/#.*$//" "%TmpGnu%\include.1.txt" > "%TmpGnu%\include.2.txt"
      rem Remove Leading/trailing whitespace from the lines that are left.
      "%binGnu%\sed.exe" -r -e "s/^[ \t]*//;s/[ \t]*$//" "%TmpGnu%\include.2.txt" > "%TmpGnu%\include.3.txt"
      copy /b "%TmpGnu%\include.3.txt" "%TmpGnu%\%IncPkgList%" > nul
      For /F "usebackq" %%a in ( "%TmpGnu%\%IncPkgList%" ) DO IF EXIST "%OldPkg%\%%a.*" move "%OldPkg%\%%a.*" "%NewPkg%"
      :OvrIncludeList

      Rem
      Rem Now Download all the Pages where the download links are for the actual files Download URLS Are located
      Rem
      Echo Please wait downloading the HTML Files with the URLS to the binaries....
      Echo %date%,%Time%,Starting Download of HTML Files with Download files list...>> "%LogGnu%"

      Echo on
      @call "%TmpGnu%\Wget-HTML.cmd"

      rem >> "%LogGnu%.2"
      @Echo off
      Echo %date%,%Time%,Finished Download of HTML Files with Download files list...>> "%LogGnu%"

      Rem
      Rem Now we need to combine then extract the file list ready for download.
      Rem
      Echo %date%,%time%,Building Download List....>> "%LogGnu%"
      Echo Building Download List....

      If exist "%TmpGnu%\MasterDlList.1.txt" del "%TmpGnu%\MasterDlList.1.txt"
      For %%a in ("%TmpHtml%\*.*") do Type "%%a" >> "%TmpGnu%\MasterDlList.1.txt"
      rem strip down to only sourceforce downloads.sourceforge.net
      "%binGnu%\sed.exe" -n "/downloads.sourceforge.net/p"  "%TmpGnu%\MasterDlList.1.txt" > "%TmpGnu%\MasterDlList.2.txt"
      rem Strip out URLS
      rem find href="
      "%binGnu%\sed.exe" -e "s/href=/\n/g" "%TmpGnu%\MasterDlList.2.txt" > "%TmpGnu%\MasterDlList.3.txt"
      rem find  onClick
      "%binGnu%\sed.exe" -e "s/onClick/\n/g" "%TmpGnu%\MasterDlList.3.txt" > "%TmpGnu%\MasterDlList.4.txt"
      rem Keep Downloads Links
      "%binGnu%\sed.exe" -n "/downloads\./p" "%TmpGnu%\MasterDlList.4.txt" > "%TmpGnu%\MasterDlList.5.txt"
      rem Pull out (bin\|doc\|lib\|dep\)\.zip
      rem "%binGnu%\sed.exe" -n "/bin\|doc\|lib\|dep\.zip/p" "%TmpGnu%\MasterDlList.5.txt" > "%TmpGnu%\MasterDlList.6.txt"
      "%binGnu%\sed.exe" -r -n "/(-bin|-doc|-lib|-dep)\.zip/p" "%TmpGnu%\MasterDlList.5.txt" > "%TmpGnu%\MasterDlList.6.txt"
      rem remove exe files 
      "%binGnu%\sed.exe" -n "/.zip/p" "%TmpGnu%\MasterDlList.6.txt" > "%TmpGnu%\MasterDlList.7.txt"
      rem remove amp;
      "%binGnu%\sed.exe" -e "s/amp\;//g" "%TmpGnu%\MasterDlList.7.txt" > "%TmpGnu%\MasterDlList.8.txt"
      rem strip "" chars
      "%binGnu%\sed.exe" -e "s/\x22//g" "%TmpGnu%\MasterDlList.8.txt" > "%TmpGnu%\MasterDlList.9.txt"
      Rem Remove file that we already have from Download list.
      copy /b "%TmpGnu%\MasterDlList.9.txt" "%TmpGnu%\MasterDlList.10.txt"
      for /F "usebackq" %%a IN ("%TmpGnu%\PkgsAlreadyGot.txt") DO "%binGnu%\sed.exe" -i -n "/%%a/!p" "%TmpGnu%\MasterDlList.10.txt"
      copy /b "%TmpGnu%\MasterDlList.10.txt" "%TmpGnu%\MasterDlList.txt"

      rem now get the files
      Echo %date%,%time%,Started Downloading the Files....>> "%LogGnu%"
      Echo Downloading the Files.... 
      rem No Directories
      rem pause
      "%binGnu%\wget.exe" %WgetOpts% -nd --directory-prefix="%NewPkg%" -c --input-file="%TmpGnu%\MasterDlList.txt"
      Echo %date%,%time%,Completed Downloading the Files....>> "%LogGnu%"
      Echo Downloading the Files....Finished. 
         
       

      :End
      ================== End Snip ==================

       
    • Doug Sweeney
      Doug Sweeney
      2008-10-16

      Here are the patches I made to my download.bat to get the download process working again. It worked for me, but YMMV...

      You should be able to cut-and-paste everything after the 'snip' line and save it into a file called 'download.diff' in the same directory where 'download.bat' resides. Then just run patch:

        patch -c download.bat download.diff

      which should apply the patch file to download.bat.

      Most of the download.bat is unchanged from the 0.6.19 version, with just the modifications I needed to get everything working again.

      ==== snip =========================
      *** download.bat.0.6.19    2007-05-18 00:27:01.344313300 -0400
      --- download.bat    2008-10-16 11:26:14.515625000 -0400
      ***************
      *** 7,12 ****
      --- 7,16 ----
        :: date   : May 17, 2007
        :: version: 0.6.19
        ::
      + :: patched: Oct 15, 2008 by DJ Sweeney, <dj.sweeney@sweeneyconcepts.com>
      + ::          Fixed to work with newer Sourceforge page structure
      + ::          NOTE: Will likely break the NEXT time the page structure changes
      + ::
        :: Copyright (c) 2007 by Mathias Michaelis, <michaelis@tcnet.ch>
        ::
        :: -------------------------------------------------------------------
      ***************
      *** 200,205 ****
      --- 204,214 ----
        :set_wgetrc
        if ".%WGETRC%"=="." set WGETRC=%CD%\bin\wget.ini
       
      + ::
      + :: If -d was specified, use existing getgnuwin32.lst to download packages
      + ::
      + if defined GNUWIN32_DIRECT_DOWNLOAD goto prepare_download_packages
      +
        ::
        :: If we are in verbose mode, tell what we're going to do
        ::
      ***************
      *** 248,278 ****
        :: file.
        ::
        if not exist project.html goto restart_download_project_site
      - if defined GNUWIN32_ALL_PROJECT_SITES goto create_project_list
        bin\sed -n "/<\/html *>/Ip" project.html >project.tmp
        bin\test -s project.tmp
        if errorlevel 1 goto restart_download_project_site
      - goto end_download_project_site
      -
        ::
      ! :: If only a part of project.html could be received, the probability is
      ! :: very high that this part contains a list of all individual gnuwin32
      ! :: projects along with the corresponding urls. So try to follow this
      ! :: links.
        ::
      - :create_project_list
        bin\sed -n "s/>/>\n/g;s/</\n</g;s/\(\s*\n\)\+\s*/\n/g;s/^\n\+//;s/\n\+$//;s/^\s\+//;s/.\+/&/p" project.html >normal.html
      ! bin\sed -n "/\bid[ \t]*=[ \t]*\(\d034frelease\d034\|\bfrelease\b\)/I,/<\/table>/Ip" normal.html >project.html
        del normal.html
        bin\sed -n "$p" project.html | bin\sed -n "/<\/table>/Ip" >project.tmp
        bin\test -s project.tmp
        if errorlevel 1 goto restart_download_project_site
      ! bin\sed -n "s/\&amp;amp;/\&amp;/Ig;s/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\)\d034>/http:\/\/sourceforge.net\1/Ip" project.html >project.tmp
        del project.html
      ! for /f "delims=" %%i in (project.tmp) do bin\wget --no-cache -O - %%i >>project.html
        goto end_download_project_site
       
        ::
        :: If the download of the whole project site failed, try to use only
        :: a part of it. This part probably allows to create a list of all
        :: gnuwin32 sub projects.
      --- 257,324 ----
        :: file.
        ::
        if not exist project.html goto restart_download_project_site
        bin\sed -n "/<\/html *>/Ip" project.html >project.tmp
        bin\test -s project.tmp
        if errorlevel 1 goto restart_download_project_site
        ::
      ! :: Revised to handle newer Sourceforge structure
      ! ::
      ! :: The main sourceforge link now only contains a browse list of each
      ! :: package and it's current release number, BUT NO file listings!
      ! ::
      ! :: To get the current version's files, we now must do essentially the
      ! :: same thing we do for the '-a' option, but only for the current release.
      ! ::
      ! :: I've changed a few things around, but the core of this file still works
      ! :: the same.
      ! ::
      ! :: Messy and slow, but it works...
        ::
        bin\sed -n "s/>/>\n/g;s/</\n</g;s/\(\s*\n\)\+\s*/\n/g;s/^\n\+//;s/\n\+$//;s/^\s\+//;s/.\+/&/p" project.html >normal.html
      ! bin\sed -n "/\bid[ \t]*=[ \t]*\(\d034yui-main\d034\|\byui-main\b\)/I,/<\/table>/Ip" normal.html >project.html
        del normal.html
        bin\sed -n "$p" project.html | bin\sed -n "/<\/table>/Ip" >project.tmp
        bin\test -s project.tmp
        if errorlevel 1 goto restart_download_project_site
      ! REM -- copy project.html project.debug.html
      ! if defined GNUWIN32_ALL_PROJECT_SITES (
      !     call :create_project_list_all
      ! ) else (
      !     call :create_project_list_current
      ! )
        del project.html
      ! echo == Building File Lists...
      ! for /f "tokens=1,2 delims= " %%i in (project.tmp) do (
      !     if defined GNUWIN32_ALL_PROJECT_SITES (
      !         echo Getting list of files [all versions]: %%i
      !     ) else (
      !         echo Getting list of files [current version]: %%i
      !     )
      !     bin\wget --quiet --progress=dot --no-cache -O - %%j >>project.html
      ! )
        goto end_download_project_site
       
        ::
      + :: Create list using current release of package
      + ::  == Called from main download section above
      + ::
      + :create_project_list_current
      + bin\sed -n "s/\&amp;amp;/\&amp;/Ig;/^<a\s\+href\s*=\s*\d034\/project\/showfiles.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\d034>/{n;s/.*/& /I;h};/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\&amp;release_id=[0-9]\+\)\d034>/{G;s/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\&amp;release_id=[0-9]\+\)\d034>\s*\(\S*\)/\2 http:\/\/sourceforge.net\1/I;p;d}" project.html >project.tmp.presort
      + :: Make sure no duplicate entries exist...
      + bin\sort -u project.tmp.presort -o project.tmp
      + goto :EOF
      +
      + ::
      + :: Create list including all available releases of package
      + ::  == Called from main download section above
      + ::
      + :create_project_list_all
      + bin\sed -n "s/\&amp;amp;/\&amp;/Ig;/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\)\d034>/{N;s/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\)\d034>\n\(\S*\)/\2 http:\/\/sourceforge.net\1/Imp}" project.html >project.tmp.presort
      + :: Make sure no duplicate entries exist...
      + bin\sort -u project.tmp.presort -o project.tmp
      + goto :EOF
      +
      + ::
        :: If the download of the whole project site failed, try to use only
        :: a part of it. This part probably allows to create a list of all
        :: gnuwin32 sub projects.
      ***************
      *** 319,324 ****
      --- 365,371 ----
        if not exist project.html goto getgnuwin32_empty
        if not "%GNUWIN32_VERBOSE%"=="" echo = Analysing file list ...
        bin\sed -n "s/>/>\n/g;s/</\n</g;s/\(\s*\n\)\+\s*/\n/g;s/^\n\+//;s/\n\+$//;s/^\s\+//;s/.\+/&/p" project.html >normal.html
      + REM -- copy normal.html normal.debug.html
        del project.html
       
        ::
      ***************
      *** 328,334 ****
        :: programmed. That's why it's a bit risky. If this script is not
        :: up-to-date, this step may fail!
        ::
      ! bin\sed -n "/^<tr\s\+class\s*=\s*""""package""""\s*>/{n;n;n;s/.*/ &/p};/^<a href=""""http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*"""">$/Is//\1/p" normal.html >getgnuwin32.tmp
        del normal.html
       
        ::
      --- 375,381 ----
        :: programmed. That's why it's a bit risky. If this script is not
        :: up-to-date, this step may fail!
        ::
      ! bin\sed -n "/^<tr\s\+class\s*=\s*""""package""""\s*>/{n;n;n;s/.*/ &/p};/^<a id=""""showfiles_download.*"""" class=""""sfx_.*"""" href=""""http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*"""" .*>$/Is//\1/p" normal.html >getgnuwin32.tmp
        del normal.html
       
        ::
      ***************
      *** 469,485 ****
      --- 516,535 ----
        bin\sed "s/  *$//;s/ /\n/g" loadlist.tmp >loadlist.txt
        del loadlist.tmp
        cd packages
      + set NO_DOWNLOADS=TRUE
        for /f %%f in (..\filelist.txt) do (
            if not exist ..\oldpacks\%%f (
                for /f %%m in (..\loadlist.txt) do (
                    if not exist %%f ..\bin\wget "%%m%%f"
                )
      +         set NO_DOWNLOADS=
            ) else (
                move /y ..\oldpacks\%%f .
            )
        )
        cd ..
        del loadlist.txt
      + if defined NO_DOWNLOADS echo No packages to download.
       
        ::
        :: Clean up things

       
    • I tried to use the patch and I get the following error:
      D:\INSTALL\GetGnuWin32>patch -c download.bat download.diff
      patch: **** malformed patch at line 6: :: date   : May 17, 2007

       
      • Doug Sweeney
        Doug Sweeney
        2008-10-20

        I tested this patch with the exact command line I listed and it worked fine on my machine. I'm not an expert on the patch command, but I can think of two possible things to check:

        1) Are you using the most current version of the 'patch' command?

        The current version is 2.5.9 and can be downloaded and installed directly from the GnuWin32 sourceforge page

        2) Are you trying to patch the 0.6.19 version of the GetGnuWin32 package?

        I have only tested this patch against the most recent download.bat (from the 0.6.19 version of GetGnuWin32).

        If you can answer 'yes' to both of these, then I don't know what to try, but I will dig deeper and try to figure it out. Like I said, I'm no expert on patch, so I might have messed something up when creating the patch.

         
    • Working version of download.bat ( patch was applied by hands :-) )
      =====Snip=====
      :: -------------------------------------------------------------------
      ::
      :: download.bat -- Maintain a small local gnuwin32 package archive
      ::                 within the cmd console of windows XP
      ::
      :: author : Mathias Michaelis <michaelis@tcnet.ch>
      :: date   : May 17, 2007
      :: version: 0.6.19
      ::
      :: patched: Oct 15, 2008 by DJ Sweeney, <dj.sweeney@sweeneyconcepts.com>
      ::          Fixed to work with newer Sourceforge page structure
      ::          NOTE: Will likely break the NEXT time the page structure changes
      ::
      :: Copyright (c) 2007 by Mathias Michaelis, <michaelis@tcnet.ch>
      ::
      :: -------------------------------------------------------------------
      ::
      :: Permission is granted to copy, distribute and/or modify this document
      :: under the terms of the GNU Free Documentation License, Version 1.2
      :: or any later version published by the Free Software Foundation;
      :: with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
      :: Texts.  A copy of the license is included in the file LICENSE.TXT
      ::
      ::
      :: Acknowledgments
      :: ---------------
      ::
      :: See file README.TXT in this directory. Thanks to all who advised me,
      :: gave me some hints, sent me some patches or encouraged me to go on!
      ::
      @echo off
      setlocal ENABLEEXTENSIONS
      set LC_ALL=C

      ::
      :: Customization
      :: -------------
      ::
      :: Select one or more of the following mirrors first! Please separate
      :: the mirrors by space, e.g. GNUWIN32_MIRROR="umn unc".
      ::
      :: aleron   (US)
      :: belnet   (BE)
      :: easynews (US)
      :: heanet   (IE)
      :: internap (US)
      :: jaist    (JP)
      :: keihanna (JP)
      :: kent     (UK)
      :: mesh     (DE)
      :: optusnet (AU)
      :: ovh      (FR)
      :: puzzle   (CH)
      :: switch   (CH)
      :: umn      (US)
      :: unc      (US)
      :: voxel    (US)
      ::
      if .%GNUWIN32_MIRROR%==. set GNUWIN32_MIRROR=umn

      ::
      :: Maximal number of trials to download the gnuwin32 project site on
      :: sourceforge.net
      ::
      set /a GNUWIN32_MAX_TRIALS=4

      ::
      :: Process command line options. This can be a list of some above
      :: mentioned mirrors and/or the following:
      ::
      :: -v: be verbose.
      :: -n: only create a new getgnuwin32.lst file, no download.
      :: -a: go through all gnuwin32 project pages when collecting the
      ::     information for the getgnuwin32.lst file.
      ::
      set GNUWIN32_CMDLINE_MIRROR="switch"

      :command_line
      if .%1==. goto command_line_end
      if "%1"=="-v" goto set_verbose
      if "%1"=="/v" goto set_verbose
      if "%1"=="-n" goto set_no_packages
      if "%1"=="/n" goto set_no_packages
      if "%1"=="-a" goto set_all_project_sites
      if "%1"=="/a" goto set_all_project_sites
      if "%1"=="-d" goto set_direct_download
      if "%1"=="/d" goto set_direct_download
      if "%1"=="-h" goto show_help_text
      if "%1"=="/h" goto show_help_text
      if "%1"=="-?" goto show_help_text
      if "%1"=="/?" goto show_help_text
      if "%1"=="--help" goto show_help_text
      if "%1"=="--version" goto show_version_text
      set GNUWIN32_CMDLINE_MIRROR=%GNUWIN32_CMDLINE_MIRROR% %1
      shift
      goto command_line

      :set_verbose
      set GNUWIN32_VERBOSE=TRUE
      shift
      goto command_line

      :set_no_packages
      set GNUWIN32_NO_PACKAGES=TRUE
      shift
      goto command_line

      :set_all_project_sites
      set GNUWIN32_ALL_PROJECT_SITES=TRUE
      shift
      goto command_line

      :set_direct_download
      set GNUWIN32_DIRECT_DOWNLOAD=TRUE
      shift
      goto command_line

      :show_help_text
      echo = download.bat (version 0.6.19^)
      echo = Copyright (c^) 2007 by Mathias Michaelis, ^<michaelis@tcnet.ch^>
      echo = This is free software; see the source for copying conditions.  There is NO
      echo = warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
      echo =
      echo = usage:
      echo =  download --version
      echo =  download --help
      echo =  download [-v] [-n] [-a] [-d]
      echo =
      echo = description:
      echo =  download.bat, started without any command line options, will try to do two
      echo =  things:
      echo =  1. To get an actual file list of all gnuwin32 projects and store it into
      echo =     the file getgnuwin32.lst. If this file exists already and there is no
      echo =     file named getgnuwin32.bak, the original getgnuwin32.lst is copied to
      echo =     getgnuwin32.bak first.
      echo =  2. The newest versions of all files listed in getgnuwin32.lst are down-
      echo =     loaded, except the files listed in include.txt, for which exactly the
      echo =     therein specified versions are downloaded, and except the files listed
      echo =     in exclude.txt.
      echo =  The behaviour of download.bat can be modified by the following options:
      echo =  -v : verbose mode. There are more lines printed on screen.
      echo =  -n : no downloads. Only step 1 is executed, step 2 is skipped.
      echo =  -a : browse all individual gnuwin32 project sites. In order to execte step
      echo =       1, normaly only the gnuwin32 main project site is used. Alas, this
      echo =       has two disadvantages: Older versions of certain files aren't listed
      echo =       there, and yet worse: Very often the download of the main project
      echo =       site fails, probably because it's too big. This is why this option
      echo =       can be used, even if it's more time consuming.
      echo =  -d : direct download. Skip step 1. This is only possible if there is a
      echo =       file list located in getgnuwin32.lst.
      echo =
      goto bye

      :show_version_text
      echo = download.bat (version 0.6.19^)
      echo = Copyright (c^) 2007 by Mathias Michaelis, ^<michaelis@tcnet.ch^>
      echo = This is free software; see the source for copying conditions.  There is NO
      echo = warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
      echo =
      goto bye

      ::
      :: Verify if command lines are meaningful
      ::
      :command_line_end
      if not defined GNUWIN32_DIRECT_DOWNLOAD goto set_gnuwin32_mirror
      if not exist getgnuwin32.lst goto no_getgnuwin32_file
      if not defined GNUWIN32_NO_PACKAGES goto set_gnuwin32_mirror
      echo = Error: -n and -d option are specified together. So I have nothing to do!
      echo =
      goto bye

      :no_getgnuwin32_file
      echo = Error: -d option was specified, but no getgnuwin32.lst file is found. Please
      echo =        get this file from somewhere or don't use the -d option to automati-
      echo =        cally try to download this file.
      echo =
      goto bye

      ::
      :: If command line options were specified, overwrite environment
      :: variables
      ::
      :set_gnuwin32_mirror
      if "%GNUWIN32_CMDLINE_MIRROR%"=="" goto environment
      set GNUWIN32_MIRROR=%GNUWIN32_CMDLINE_MIRROR:~1%
      set GNUWIN32_LOAD=
      set GNUWIN32_CMDLINE_MIRROR=

      ::
      :: After the mirrors are known, the download address of that mirrors
      :: can be constructed (if not already done):
      ::
      :environment
      if not .%GNUWIN32_LOAD%==. goto set_wgetrc
      set GNUWIN32_LOAD=http://%GNUWIN32_MIRROR:"=%.dl.sourceforge.net/sourceforge/gnuwin32/
      set GNUWIN32_LOAD=%GNUWIN32_LOAD: =.dl.sourceforge.net/sourceforge/gnuwin32/ http://%

      ::
      :: Perhaps the user has specified her/his own .wgetrc file by means
      :: of the WGETRC environment variable. If not, then use the one in
      :: the local bin directory
      ::
      :set_wgetrc
      if ".%WGETRC%"=="." set WGETRC=%CD%\bin\wget.ini

      :: 
      :: If -d was specified, use existing getgnuwin32.lst to download packages
      ::
      if defined GNUWIN32_DIRECT_DOWNLOAD goto prepare_download_packages

      ::
      :: If we are in verbose mode, tell what we're going to do
      ::
      if "%GNUWIN32_VERBOSE%"=="" goto start_download_project_site

      echo =
      echo = URL of file list:
      echo =  https://sourceforge.net/project/showfiles.php?group_id=23617
      echo =
      echo = List of mirrors (high priority first):
      echo =  %GNUWIN32_MIRROR%
      echo =
      echo = List of URLs (high priority first):
      echo %GNUWIN32_LOAD% >loadlist.txt
      bin\sed "s/  *$//;s/.*/=  &/;s/[^ =] /\n=  /g" loadlist.txt
      echo =
      del loadlist.txt

      ::
      :: First, get the file list form the official project site(s) on sourceforge.net
      ::
      :start_download_project_site
      if not "%GNUWIN32_VERBOSE%"=="" echo = Downloading file list from sourceforge.net ...
      if exist project.html del project.html
      set /a GNUWIN32_TRIALS=0

      :restart_download_project_site
      if /i %GNUWIN32_TRIALS% GEQ %GNUWIN32_MAX_TRIALS% goto stop_download_project_site
      set /a GNUWIN32_RATE="250 / (GNUWIN32_TRIALS %% 10 + 1)"
      set /a GNUWIN32_TRIALS="GNUWIN32_TRIALS + 1"
      set /a GNUWIN32_ODD_TRIAL="GNUWIN32_TRIALS & 1"
      if /i %GNUWIN32_TRIALS% GTR 1 echo =
      if /i %GNUWIN32_TRIALS% EQU 2 echo = Download project.html failed. 2nd trial ...
      if /i %GNUWIN32_TRIALS% EQU 3 echo = Download project.html failed. 3rd trial ...
      if /i %GNUWIN32_TRIALS% GTR 3 echo = Download project.html failed. %GNUWIN32_TRIALS%th trial ...
      if /i %GNUWIN32_TRIALS% GTR 1 echo = (Limit download rate to %GNUWIN32_RATE%kB)
      if /i %GNUWIN32_TRIALS% GTR 1 echo =
      if /i %GNUWIN32_ODD_TRIAL% EQU 1 (
          bin\wget --ca-certificate=bin\cacert.pem --no-cache --limit-rate=%GNUWIN32_RATE%k -O project.html https://sourceforge.net/project/showfiles.php?group_id=23617
      ) else (
          bin\wget --no-cache --limit-rate=%GNUWIN32_RATE%k -O project.html http://sourceforge.net/project/showfiles.php?group_id=23617
      )

      ::
      :: This download very often fails. So check if we've got the whole
      :: file.
      ::
      if not exist project.html goto restart_download_project_site
      bin\sed -n "/<\/html *>/Ip" project.html >project.tmp
      bin\test -s project.tmp
      if errorlevel 1 goto restart_download_project_site
      ::
      :: Revised to handle newer Sourceforge structure
      ::
      :: The main sourceforge link now only contains a browse list of each
      :: package and it's current release number, BUT NO file listings!
      :: 
      :: To get the current version's files, we now must do essentially the 
      :: same thing we do for the '-a' option, but only for the current release.
      ::
      :: I've changed a few things around, but the core of this file still works
      :: the same.
      ::
      :: Messy and slow, but it works...
      ::
      bin\sed -n "s/>/>\n/g;s/</\n</g;s/\(\s*\n\)\+\s*/\n/g;s/^\n\+//;s/\n\+$//;s/^\s\+//;s/.\+/&/p" project.html >normal.html
      bin\sed -n "/\bid[ \t]*=[ \t]*\(\d034yui-main\d034\|\byui-main\b\)/I,/<\/table>/Ip" normal.html >project.html
      del normal.html
      bin\sed -n "$p" project.html | bin\sed -n "/<\/table>/Ip" >project.tmp
      bin\test -s project.tmp
      if errorlevel 1 goto restart_download_project_site
      REM -- copy project.html project.debug.html
      if defined GNUWIN32_ALL_PROJECT_SITES (
          call :create_project_list_all
      ) else (
          call :create_project_list_current
      )
      del project.html
      echo == Building File Lists...
      for /f "tokens=1,2 delims= " %%i in (project.tmp) do (
          if defined GNUWIN32_ALL_PROJECT_SITES (
              echo Getting list of files [all versions]: %%i
          ) else (
              echo Getting list of files [current version]: %%i
          )
          bin\wget --quiet --progress=dot --no-cache -O - %%j >>project.html
      )
      goto end_download_project_site

      ::
      :: Create list using current release of package
      ::  == Called from main download section above
      ::
      :create_project_list_current
      bin\sed -n "s/\&amp;amp;/\&amp;/Ig;/^<a\s\+href\s*=\s*\d034\/project\/showfiles.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\d034>/{n;s/.*/& /I;h};/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\&amp;release_id=[0-9]\+\)\d034>/{G;s/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\&amp;release_id=[0-9]\+\)\d034>\s*\(\S*\)/\2 http:\/\/sourceforge.net\1/I;p;d}" project.html >project.tmp.presort
      :: Make sure no duplicate entries exist...
      bin\sort -u project.tmp.presort -o project.tmp
      goto :EOF

      ::
      :: Create list including all available releases of package
      ::  == Called from main download section above
      ::
      :create_project_list_all
      bin\sed -n "s/\&amp;amp;/\&amp;/Ig;/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\)\d034>/{N;s/<a[ \t]\+href[ \t]*=[ \t]*\d034\(\/project\/showfiles\.php?group_id=[0-9]\+\&amp;package_id=[0-9]\+\)\d034>\n\(\S*\)/\2 http:\/\/sourceforge.net\1/Imp}" project.html >project.tmp.presort
      :: Make sure no duplicate entries exist...
      bin\sort -u project.tmp.presort -o project.tmp
      goto :EOF

      ::
      :: If the download of the whole project site failed, try to use only
      :: a part of it. This part probably allows to create a list of all
      :: gnuwin32 sub projects.
      ::
      :stop_download_project_site
      if defined GNUWIN32_ALL_PROJECT_SITES goto ultimate_stop_download_project_site
      echo =
      echo = Warning: Download failed %GNUWIN32_MAX_TRIALS% times. We try no to browse
      echo = all individual gnuwin32 project sites instead of the main project site.
      echo =
      set GNUWIN32_ALL_PROJECT_SITES=TRUE
      set /a GNUWIN32_MAX_TRIALS="GNUWIN32_MAX_TRIALS + 4"
      if not exist project.html goto restart_download_project_site
      goto create_project_list

      ::
      :: If the download failed many times, try to use an already present
      :: getgnuwin32.lst
      ::
      :ultimate_stop_download_project_site
      if exist project.tmp del project.tmp
      if exist project.html del project.html
      if defined GNUWIN32_NO_PACKAGES goto failed_create_filelist
      echo =
      echo = Warning: Couldn't download the gnuwin32 project site from sourceforge.net. We
      echo = try to use an already present file list now. Note that this list must reside
      echo = in a file named getgnuwin32.lst.
      echo =
      goto prepare_download_packages

      :failed_create_filelist
      echo = Error: Couldn't download the gnuwin32 project site from sourceforge.net. So
      echo = a new file list can't be created. Try to restart this program when the server
      echo = at sourceforge.net is less busy.
      echo =
      goto bye

      ::
      :: If the gnuwin32 project site could be downloaded, normalize it. Every
      :: tag has to be in its own lins. Empty lines are removed.
      ::
      :end_download_project_site
      del project.tmp
      if not exist project.html goto getgnuwin32_empty
      if not "%GNUWIN32_VERBOSE%"=="" echo = Analysing file list ...
      bin\sed -n "s/>/>\n/g;s/</\n</g;s/\(\s*\n\)\+\s*/\n/g;s/^\n\+//;s/\n\+$//;s/^\s\+//;s/.\+/&/p" project.html >normal.html
      REM -- copy normal.html normal.debug.html
      del project.html

      ::
      :: Convert the normalized HTML file in plain text. Take only the
      :: information that is relevant here: The package list.
      :: This procedure depends very much on the way the HTML file is
      :: programmed. That's why it's a bit risky. If this script is not
      :: up-to-date, this step may fail!
      ::
      bin\sed -n "/^<tr\s\+class\s*=\s*""""package""""\s*>/{n;n;n;s/.*/ &/p};/^<a id=""""showfiles_download.*"""" class=""""sfx_.*"""" href=""""http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*"""" .*>$/Is//\1/p" normal.html >getgnuwin32.tmp
      del normal.html

      ::
      :: Before we continue, we have to check if getgnuwin32.tmp contains some
      :: information. If yes, old file lists must be removed first.
      ::
      bin\test -s getgnuwin32.tmp
      if not errorlevel 1 goto create_new_filelist
      goto getgnuwin32_empty

      :getgnuwin32_empty
      echo =
      echo = Warning: Couldn't completely download the gnuwin32 project site from
      echo = sourceforge.net, or the downloaded file has not the format that is
      echo = expected here -- maybe this script is out of date. We try now to use
      echo = an already existing file list. This file list must reside in a file
      echo = named getgnuwin32.lst.
      echo =
      if exist getgnuwin32.tmp del getgnuwin32.tmp
      goto prepare_download_packages

      ::
      :: The file getgnuwin32.lst is ultimately used at this place
      ::
      :prepare_download_packages
      if exist getgnuwin32.lst goto clean_old_lists
      echo = Error: getgnuwin32.lst doesn't exist. Possibly the above download of the
      echo = gnuwin32 project web site on sourceforge.net failed. You have now two
      echo = possibilities:
      echo =
      echo = 1. Try to restart download.bat later, if possible at a time where the
      echo =    server at sourceforge.net is less busy.
      echo =
      echo = 2. Ask a friend at another place to run download.bat with the -n option.
      echo =    This will create getgnuwin32.lst, the file you need to continue. If
      echo =    you have it, run download.bat with the -d option, that will direct
      echo =    download the desired packages and skip the download of the gnuwin32
      echo =    project site.
      echo =
      goto bye

      ::
      :: Create a new getgnuwin32.lst and backup the old one
      ::
      :create_new_filelist
      if exist getgnuwin32.lst if not exist getgnuwin32.bak ren getgnuwin32.lst getgnuwin32.bak
      if exist getgnuwin32.lst del getgnuwin32.lst
      ren getgnuwin32.tmp getgnuwin32.lst
      if defined GNUWIN32_NO_PACKAGES goto bye
      goto clean_old_lists

      ::
      :: Clean up old file lists etc.
      ::
      :clean_old_lists
      if exist filelist.txt del filelist.txt
      if exist packages.txt del packages.txt

      ::
      :: Extract the filelist from temporary file filelist.tmp
      ::
      :extract_filelist
      bin\sed "/^ /d" getgnuwin32.lst >filelist.txt

      ::
      :: Assume: filelist.txt is already sorted by filenames and fileversion
      :: so that new versions are listed first.
      ::
      :: Now create a package list from the files with the newest versions
      ::
      :: bin\sed -n "/^[^ ]/s/\(.*\)-...\.zip/\1/p" filelist.txt >packages.tmp
      :: bin\sed ":x;N;s/^\(\([^-]*-[^[:digit:]]\)*[^-]*\)\(-[[:digit:]].*\)\?\n\1\(-[[:digit:]].*\)\?$/\1\3/;tx;P;D" packages.tmp >packages.txt
      ::
      :: New: getgnuwin32.lst contains also package information. This simplifies
      :: the process of generating a package list.
      ::
      bin\sed -n "/^ /{n;s/\(.*\)-...\.zip/\1/p}" getgnuwin32.lst >packages.txt

      ::
      :: Cancel all packages which names are listed within the include.txt and
      :: the exclude.txt files.
      ::
      if exist include.txt bin\sed -n "s/\s*#.*$//;tx;:x;s/^\s*$//;t;s/\s*//g;p" include.txt >include.tmp
      if exist include.tmp bin\sed "s/^\(\([^-]*-[^[:digit:]]\)*[^-]*\).*$/\1 -/" include.tmp >exclude.tmp
      if exist exclude.txt bin\sed -n "s/\s*#.*$//;tx;:x;s/^\s*$//;t;s/^\s*\([^[:space:]]*\)\s*$/\1 -/p" exclude.txt >>exclude.tmp
      if not exist exclude.tmp goto add_include

      bin\sort -u packages.txt exclude.tmp -o packages.tmp
      bin\sed "/^.* -$/{:x;N;s/^\([^\n]*\) -\n\1-[[:digit:]].*$/\1 -/I;tx;D}" packages.tmp >packages.txt
      del packages.tmp
      del exclude.tmp

      ::
      :: Add packages in include.txt
      ::
      :add_include
      if not exist include.tmp goto shuffle
      copy include.tmp+packages.txt packages.tmp >nul
      del packages.txt
      del include.tmp
      ren packages.tmp packages.txt

      ::
      :: Shuffle package list and file list
      ::
      :shuffle
      bin\sed "s/^.*[^-]$/&-/" packages.txt >packages.tmp
      bin\sort filelist.txt packages.tmp -o filelist.tmp
      del packages.tmp

      ::
      :: Cancel all files which don't belong to a package
      ::
      bin\sed -n "h;N;s/^\([^\n]\+\)\n\1/\1/;tx;D;:x;G;P;D" filelist.tmp >filelist.txt
      del filelist.tmp

      ::
      :: Download the desired files from the server
      ::
      :download_packages
      if defined GNUWIN32_NO_PACKAGES goto bye
      if not "%GNUWIN32_VERBOSE%"=="" echo = Starting download of all files found within the file list ...
      if not exist packages goto makepackages
      if not exist oldpacks goto makeoldpacks

      move /y packages\*.* oldpacks >nul
      rmdir /s /q packages
      goto makepackages

      :makeoldpacks
      ren packages oldpacks

      :makepackages
      mkdir packages

      :download
      echo %GNUWIN32_LOAD% >loadlist.tmp
      bin\sed "s/  *$//;s/ /\n/g" loadlist.tmp >loadlist.txt
      del loadlist.tmp
      cd packages
      set NO_DOWNLOADS=TRUE
      for /f %%f in (..\filelist.txt) do (
          if not exist ..\oldpacks\%%f (
              for /f %%m in (..\loadlist.txt) do (
                  if not exist %%f ..\bin\wget "%%m%%f"
              )
              set NO_DOWNLOADS=
          ) else (
              move /y ..\oldpacks\%%f .
          )
      )
      cd ..
      del loadlist.txt
      if defined NO_DOWNLOADS echo No packages to download.

      ::
      :: Clean up things
      ::
      if not exist oldpacks goto bye
      dir /b oldpacks >oldpacks.txt
      bin\test -s oldpacks.txt
      if errorlevel 1 rmdir /s /q oldpacks
      if exist oldpacks.txt del oldpacks.txt

      :bye
      endlocal

      =====Snip End===== 

       
    • Gi-Jo Son
      Gi-Jo Son
      2008-11-15

      I did just drag between Snip.

      paste and run patch.exe equal error occured.

      1. I found some <space> and <tab> was skipped.
      2. each line have one <space>, end of line.

      example:
      ':: date : May 17, 2007 ' (before)
      '  :: date   : May 17, 2007' (after)
      (add two <space> on front and add two <space> between "data" and ":")

      ':: Copyright (c) 2007 by Mathias Michaelis, <michaelis@tcnet.ch> ' (before)
      '  :: Copyright (c) 2007 by Mathias Michaelis, <michaelis@tcnet.ch>' (after)
      (add two <space> on front)

      ':set_wgetrc ' (before)
      '  :set_wgetrc' (after)
      (add two <space> on front)

      'bin\sed "s/ *$//;s/ /\n/g" loadlist.tmp >loadlist.txt ' (before)
      '  bin\sed "s/  *$//;s/ /\n/g" loadlist.tmp >loadlist.txt' (after)
      (add two <space> on front and add one <space> between "s/" and "*$")

      'if not exist ..\oldpacks\%%f ( '            (before)
      '      if not exist ..\oldpacks\%%f ('     (after)
      (add two <space> on front and add one <tab> before "if")

      corrected diff file has no errors with patch.

      ps. Thanks Doug Sweeney and George_Saveliev.
            Without modified download.bat I couldn't install gnuwin32 packages.

       
    • Gi-Jo Son
      Gi-Jo Son
      2008-11-15

      Oops, posted contents has same problem.
      two <space> modified one <space> and <tab> disappeared.

      additional example is here.

      ':: date : May 17, 2007 ' (before)
      '<sp><sp>:: date <sp><sp>: May 17, 2007' (after)
      (add two <space> on front and add two <space> between "data" and ":")

      ':: Copyright (c) 2007 by Mathias Michaelis, <michaelis@tcnet.ch> ' (before)
      '<sp><sp>:: Copyright (c) 2007 by Mathias Michaelis, <michaelis@tcnet.ch>' (after)
      (add two <space> on front)

      ':set_wgetrc ' (before)
      '<sp><sp>:set_wgetrc' (after)
      (add two <space> on front)

      'bin\sed "s/ *$//;s/ /\n/g" loadlist.tmp >loadlist.txt ' (before)
      '<sp><sp>bin\sed "s/<sp><sp>*$//;s/ /\n/g" loadlist.tmp >loadlist.txt' (after)
      (add two <space> on front and add one <space> between "s/" and "*$")

      'if not exist ..\oldpacks\%%f ( ' (before)
      '<sp><sp><tab>if not exist ..\oldpacks\%%f (' (after)
      (add two <space> on front and add one <tab> before "if")

       
      • Doug Sweeney
        Doug Sweeney
        2008-11-18

        Looks like it got messed up when I posted it in my original message -- for some reason the leading spaces/tabs got removed. Bummer...

        I went back and looked at my local DIFF file and the spaces/tabs are there. I've placed the original diff file on my web site in a ZIP file for download.

        http://www.sweeneyconcepts.com/files/download-diff.zip

        Doug

         
    • Andy Yeung
      Andy Yeung
      2008-11-20

      I tried to use getgnuwin32 yesterday and find that it failed to to the change in sourceforge.  Before I found this thread, I tried to do some quick fix on the download.bat.  I works for me and so I post the changes I made to the download.bat here for reference.  Thanks.

      andy

      --- download.old    2008-11-20 11:21:28.000000000 +0800
      +++ download.bat    2008-11-20 12:18:33.000000000 +0800
      @@ -52,7 +52,8 @@
      :: unc      (US)
      :: voxel    (US)
      ::
      -if .%GNUWIN32_MIRROR%==. set GNUWIN32_MIRROR=umn
      +:: if .%GNUWIN32_MIRROR%==. set GNUWIN32_MIRROR=umn
      +if .%GNUWIN32_MIRROR%==. set GNUWIN32_MIRROR=kent

      ::
      :: Maximal number of trials to download the gnuwin32 project site on
      @@ -328,9 +329,20 @@
      :: programmed. That's why it's a bit risky. If this script is not
      :: up-to-date, this step may fail!
      ::
      -bin\sed -n "/^<tr\s\+class\s*=\s*""""package""""\s*>/{n;n;n;s/.*/ &/p};/^<a href=""""http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*"""">$/Is//\1/p" normal.html >getgnuwin32.tmp
      +:: bin\sed -n "/^<tr\s\+class\s*=\s*""""package""""\s*>/{n;n;n;s/.*/ &/p};/^<a href=""""http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*"""">$/Is//\1/p" normal.html >getgnuwin32.tmp
      +bin\sed -n "/^<tbody>/,/^<\/tbody>/ {/tbody/ !{/^<tr/ {n;n;n;s/.*/ &\,/;h;n;n;n;n;s/^<a href=""""\(.*\).*"""">/\&quot;http:\/\/sourceforge.net\1\&quot;/;s/\&amp;amp\;/\&amp;/g;H;x;s/\n/ /p}}}" normal.html > new.txt
      del normal.html

      +if exist getgnuwin32.tmp del getgnuwin32.tmp
      +for /f "tokens=1,2 delims=," %%f in (new.txt) do (
      +    bin\wget -O new.html %%g
      +    bin\sed -n "s/>/>\n/g; s/</\n</g; s/\(\s*\n\)\+\s*/\n/g; s/^\n\+//; s/\n\+$//;s/^\s\+//; s/.\+/&/p" new.html >new2.html
      +    bin\sed -n "/^<tr\s\+class\s*=\s*""""package""""\s*>/ {n;n;n;s/.*/ &/p}; /.*href=""""http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*"""">$/Is//\1/p" new2.html >>getgnuwin32.tmp
      +)
      +if exist new.html del new.html
      +if exist new2.html del new2.html
      +if exist new.txt del new.txt
      +
      ::
      :: Before we continue, we have to check if getgnuwin32.tmp contains some
      :: information. If yes, old file lists must be removed first.

       
    • Russ Gibson
      Russ Gibson
      2008-12-18

      Using the downloaded patch from the link above worked perfectly for me.  One thing I did do though, was I had to grab the GnuWin32 patch package and use it, as the MingW/MSYS version was too out of date.

       
    • Seems to be broken again?

       
    • Ross Smith II
      Ross Smith II
      2009-04-05

      I got download.bat working again by applying the patch I wrote at http://smithii.com/files/getgnuwin32/download.diff:

      --- download.bat.orig    2009-02-19 20:00:54.972302400 -0800
      +++ download.bat    2009-04-05 12:20:28.953125000 -0700
      @@ -340,7 +340,7 @@
      :: OUTPUT PACKAGE TABLES TO normal.tmp
      bin\sed -T -n "/<tr\s\+class\s*=\s*\d034package\d034\s*>/I,/<\/table>/Ip" normal.html > normal.tmp
      :: OUTPUT PACKAGE INFORMATION TO getgnuwin32.tmp
      -bin\sed -T -n "/<tr\s\+class\s*=\s*\d034package\d034\s*>/{n;n;n;s/.*/ &/p};/<a[ \t]\+.*href[ \t]*=[ \t]*\d034http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*\d034>$/Is//\1/p" normal.tmp >getgnuwin32.tmp
      +bin\sed -T -n "s/.*http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*$/\1/p" normal.tmp >getgnuwin32.tmp
      del normal.html normal.tmp

      ::
      @@ -403,7 +403,7 @@
      :: Extract the filelist from temporary file filelist.tmp
      ::
      :extract_filelist
      -bin\sed -T "/^ /d" getgnuwin32.lst >filelist.txt
      +bin\sort getgnuwin32.lst >filelist.txt

      ::
      :: Assume: filelist.txt is already sorted by filenames and fileversion
      @@ -417,7 +417,7 @@
      :: New: getgnuwin32.lst contains also package information. This simplifies
      :: the process of generating a package list.
      ::
      -bin\sed -T -n "/^ /{n;s/\(.*\)-...\.zip/\1/p}" getgnuwin32.lst >packages.txt
      +bin\sed -T -n "s/^\(.*\)-...\.zip/\1/p" getgnuwin32.lst >packages.txt

      ::
      :: Cancel all packages which names are listed within the include.txt and

      The patched download.bat is available here:

      http://smithii.com/files/getgnuwin32/download.bat or
      http://smithii.com/files/getgnuwin32/download.bat.txt if you want to view it in your browser.

      To fix your installation, type the following (replace "C:\Program Files\GnuWin32" with the directory you installed GnuWin32 into):

      cd /d "C:\Program Files\GnuWin32"
      set WGETRC=%CD%\bin\wget.ini
      bin\wget http://smithii.com/files/getgnuwin32/download.bat

      and you should be all set.

       
    • rmccullough
      rmccullough
      2009-04-08

      rossta,

      Your download.bat at http://smithii.com/files/getgnuwin32/download.bat does not seem to obey my exclude.txt items. Here is an example of what my exclude file looks like:
      # ----------------------------------------------------
      #
      # exclude.txt -- Packages not wanted in any version
      #                (unless overruled by include.txt)
      #
      # ----------------------------------------------------
      #
      cpp             # Documentation only. Files should be
                      # extracted to doc\cpp, but they aren't.
              # So special treatment is needed.
      fileutils       # Is now included within coreutils
      gdlib           # <gdlib> is the ancestor of <gd>
      libc            # Belongs to obsolet <glibc-doc-2.2.4>
      libio           # Belongs to obsolet <gcc-doc-2.95.2>
      libm            # Belongs to obsolet <gcc-doc-2.95.2>
      libttf          # Belongs to obsolet <freetype-1.4>
      regex-spencer   # We have already <regex>. Spencer colides
                      #  with the <file-4.13> package.
      sh-utils        # Is now included within coreutils
      standards       # Belongs to obsolet <gcc-doc-2.95.2>
      stat            # Is now included within coreutils
      textutils       # Is now included within coreutils

      # ----------------------------------------------------
      #
      # My Excludes
      #
      # ----------------------------------------------------

      a2ps
      ascii_chart
      autoconf
      automake
      barcode
      bc
      bison
      bm2font

      Yet, download.bat is not removing these from packages.txt, so they get downloaded into my packages directory. Thoughts?

       
      • Ramon Klein
        Ramon Klein
        2009-06-04

        Your script has a single flaw (which it showed at least at my system)
        In line 343 the regularexpression does not match if remove a superflus questionmark.

        342   :: OUTPUT PACKAGE INFORMATION TO getgnuwin32.tmp
        343    bin\sed -T -n "s/.*http:\/\/downloads.sourceforge.net\/gnuwin32\/\(.*-\(bin\|doc\|lib\|dep\)\.zip\)?.*$/\1/p" normal.tmp >getgnuwin32.tmp

        I had to remove the last ? then it worked perfectly.

         
      • Ross Smith II
        Ross Smith II
        2009-06-24

        rmccullough,

        My patch only changes three lines in download.bat, and none of them relate to the exclude.txt or include.txt files, so my guess is the bug is in the original download.bat file.

         
    • Nathan Beyer
      Nathan Beyer
      2009-04-09

      Have the suggested changes been posted back, so that an update can be made to the main distribution?

       
    • This is my getGnuWin32.cmd:

      @echo off
      setlocal
        set BASE_URL=http://sourceforge.net
        set GROUP_ID=23617
        set GROUP_URL=%BASE_URL%/project/showfiles.php?group_id=%GROUP_ID%

          if exist packages. (
              if not exist packages_old. mkdir packages_old
          move packages\*.* packages_old
          )
        for /f "usebackq" %%p in (`wget -qO - "%GROUP_URL%" ^| grep -Po "/project/showfiles.*release_id=\d+" ^| uniq ^| sed "s/amp;//g"`) do (
          rem wget -qO - "%BASE_URL%%%p" | grep -Po "http://downloads[^']+\.zip" | grep -Pv ".*-src\.zip" | wget -NP packages -i -
              for /f "usebackq delims=/ tokens=4" %%f in (`wget -qO - "%BASE_URL%%%p" ^| grep -Po "http://downloads[^']+\.zip" ^| grep -Pv ".*-src\.zip"`) do (
                  if exist packages_old\%%f (
                      move packages_old\%%f packages
                  ) else (
                      wget -NP packages "http://downloads.sourceforge.net/gnuwin32/%%f"
                  )
              )
        )
      endlocal

       
    • Patrick Schulz
      Patrick Schulz
      2009-06-12

      Hey guys,

      doesn't it makes sense to deploy the final scripts to the patches area?
      I would also say it is time for a new release.

      @michaelis: Any plans?

      Cheers,
      Patrick

       
    • Ross Smith II
      Ross Smith II
      2009-06-24

       
  • Jay Satiro
    Jay Satiro
    2009-10-01

    I am releasing a test build of GetGnuWin32. You can download it here:

      : http://drop.io/getgnuwin32

    The build is working as of October 01, 2009.

    sha1sum:

    3d3f2c883149a87cd3b195d29ed6b177eefb19e6 *GetGnuWin32-100109-TEST.zip

     
1 2 > >> (Page 1 of 2)