Menu
▾
▴
fink-core
|
From: Alexander H. <ale...@gm...> - 2011-09-26 21:04:28
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I asked SF.net about how to replace our ailing mirror list with something that still contains geographic information and here's the response. It looks like we'd need to do some additional coding to implement it. - -------- Original Message -------- Subject: Re: Best option to select a particular mirror for wget/curl download Date: Mon, 26 Sep 2011 16:54:45 -0400 From: SourceForge.net Support <sfn...@ge...> To: Alexander Hansen <ale...@gm...> Hello, We recommend that you simply use the "use_mirror" option to specify a download mirror. Here's an example using wget: ~~~~~~ $ wget http://sourceforge.net/projects/fink/files/fink/0.31.2/fink-0.31.2.tar.gz/download?use_mirror=garr - --2011-09-26 16:48:40-- http://sourceforge.net/projects/fink/files/fink/0.31.2/fink-0.31.2.tar.gz/download?use_mirror=garr Resolving sourceforge.net... 216.34.181.60 Connecting to sourceforge.net|216.34.181.60|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://downloads.sourceforge.net/project/fink/fink/0.31.2/fink-0.31.2.tar.gz?r=&ts=1317070120&use_mirror=garr [following] - --2011-09-26 16:48:40-- http://downloads.sourceforge.net/project/fink/fink/0.31.2/fink-0.31.2.tar.gz?r=&ts=1317070120&use_mirror=garr Resolving downloads.sourceforge.net... 216.34.181.59 Connecting to downloads.sourceforge.net|216.34.181.59|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://garr.dl.sourceforge.net/project/fink/fink/0.31.2/fink-0.31.2.tar.gz [following] - --2011-09-26 16:48:41-- http://garr.dl.sourceforge.net/project/fink/fink/0.31.2/fink-0.31.2.tar.gz Resolving garr.dl.sourceforge.net... 193.206.140.34, 2001:760:ffff:b0::34 Connecting to garr.dl.sourceforge.net|193.206.140.34|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1339174 (1.3M) [application/x-gzip] Saving to: `fink-0.31.2.tar.gz' 100%[==============================================================================>] 1,339,174 340K/s in 4.2s 2011-09-26 16:48:46 (313 KB/s) - `fink-0.31.2.tar.gz' saved [1339174/1339174] $ ~~~~~~ garr is actually almost across the globe from me, and I just used that to demonstrate that it is indeed picking a mirror it wouldn't have auto-selected (an auto-select would probably have selected voxel for me). You can see a list of mirror shortnames to use here: http://sourceforge.net/apps/trac/sourceforge/wiki/Mirrors Also note that this download URL will indeed require a tool that follows 302 redirects, wget will do that automatically, curl requires the -L option. I'm not familiar with axel, so I don't know if that needs any special consideration or not. For further reading, we have some info in our community docs on downloading via command-line here: http://sourceforge.net/p/forge/community-docs/downloading-via-cli/ Regards, Chris Tsai SourceForge Support sfn...@ge... On Mon, Sep 26, 2011 at 2:11 PM, Alexander Hansen <ale...@gm...> wrote: > Hi. > > I'm a developer with the Fink project. A good of the packages we > maintain are hosted at SourceForge, but we're currently relying on > redirection of our crufty URLs to find them. :-) We use tools > like wget, curl, axel, .. to download the source files. > > The current syntax we're using to pick out individual mirrors is, > for example, to download directly from: > > http://voxel.dl.sourceforge.net/sourceforge/ > > followed by the path to the project's source. These aren't > resolving anymore and so users have to go through redirects to get > the sources. > > Do we have the option to choose a definite mirror, e.g. so our > users can pick one close to them geographically? > ==== This e- mail message is intended only for the named recipient(s) above. It may contain confidential and privileged information. If you are not the intended recipient you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. Thank you. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6A6NAACgkQB8UpO3rKjQ85lwCffEAJE4rlSvnnrjaiaJ15X7Vs w4wAn2baxMeqLkS2SDpMOXPRaXGSNiix =Q8P2 -----END PGP SIGNATURE----- |
|
From: Alexander H. <ale...@gm...> - 2011-09-26 23:06:34
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 9/26/11 5:04 PM, Alexander Hansen wrote: > I asked SF.net about how to replace our ailing mirror list with > something that still contains geographic information and here's > the response. It looks like we'd need to do some additional coding > to implement it. I've also been playing around with Using "http://downloads.sourceforge.net/project/<project path>". This works with wget, but not curl: wget -U 'fink/0.31.2' --verbose --passive-ftp -O multicore-0.2.15.tar.gz http://downloads.sourceforge.net/project/octave/Octave\ Forge\ Packages/R2009-06-07/multicore-0.2.15.tar.gz - --2011-09-26 18:56:43-- http://downloads.sourceforge.net/project/octave/Octave%20Forge%20Packages/R2009-06-07/multicore-0.2.15.tar.gz Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 216.34.181.59 Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|216.34.181.59|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://superb-dca2.dl.sourceforge.net/project/octave/Octave%20Forge%20Packages/R2009-06-07/multicore-0.2.15.tar.gz [following] - --2011-09-26 18:56:44-- http://superb-dca2.dl.sourceforge.net/project/octave/Octave%20Forge%20Packages/R2009-06-07/multicore-0.2.15.tar.gz Resolving superb-dca2.dl.sourceforge.net (superb-dca2.dl.sourceforge.net)... 209.61.193.20 Connecting to superb-dca2.dl.sourceforge.net (superb-dca2.dl.sourceforge.net)|209.61.193.20|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 30842 (30K) [application/x-gzip] Saving to: ?multicore-0.2.15.tar.gz? 100%[======================================>] 30,842 28.6K/s in 1.1s 2011-09-26 18:56:45 (28.6 KB/s) - ?multicore-0.2.15.tar.gz? saved [30842/30842] curl --connect-timeout 30 -f -L -A 'fink/0.31.2' -O http://downloads.sourceforge.net/project/octave/Octave\ Forge\ Packages/R2009-06-07/multicore-0.2.15.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- - --:--:-- 0 curl: (22) The requested URL returned error: 400 ### execution of curl failed, exit code 22 Downloading the file "multicore-0.2.15.tar.gz" failed. > > -------- Original Message -------- Subject: Re: Best option to > select a particular mirror for wget/curl download Date: Mon, 26 Sep > 2011 16:54:45 -0400 From: SourceForge.net Support > <sfn...@ge...> To: Alexander Hansen > <ale...@gm...> > > Hello, > > We recommend that you simply use the "use_mirror" option to specify > a download mirror. Here's an example using wget: > > ~~~~~~ $ wget > http://sourceforge.net/projects/fink/files/fink/0.31.2/fink-0.31.2.tar.gz/download?use_mirror=garr > > - --2011-09-26 16:48:40-- > http://sourceforge.net/projects/fink/files/fink/0.31.2/fink-0.31.2.tar.gz/download?use_mirror=garr > > Resolving sourceforge.net... 216.34.181.60 > Connecting to sourceforge.net|216.34.181.60|:80... connected. HTTP > request sent, awaiting response... 302 Found Location: > http://downloads.sourceforge.net/project/fink/fink/0.31.2/fink-0.31.2.tar.gz?r=&ts=1317070120&use_mirror=garr > > [following] > --2011-09-26 16:48:40-- > http://downloads.sourceforge.net/project/fink/fink/0.31.2/fink-0.31.2.tar.gz?r=&ts=1317070120&use_mirror=garr > > Resolving downloads.sourceforge.net... 216.34.181.59 > Connecting to downloads.sourceforge.net|216.34.181.59|:80... > connected. HTTP request sent, awaiting response... 302 Found > Location: > http://garr.dl.sourceforge.net/project/fink/fink/0.31.2/fink-0.31.2.tar.gz > > [following] > --2011-09-26 16:48:41-- > http://garr.dl.sourceforge.net/project/fink/fink/0.31.2/fink-0.31.2.tar.gz > > Resolving garr.dl.sourceforge.net... 193.206.140.34, 2001:760:ffff:b0::34 > Connecting to garr.dl.sourceforge.net|193.206.140.34|:80... > connected. HTTP request sent, awaiting response... 200 OK Length: > 1339174 (1.3M) [application/x-gzip] Saving to: > `fink-0.31.2.tar.gz' > > 100%[==============================================================================>] > > 1,339,174 340K/s in 4.2s > > 2011-09-26 16:48:46 (313 KB/s) - `fink-0.31.2.tar.gz' saved > [1339174/1339174] > > $ ~~~~~~ > > garr is actually almost across the globe from me, and I just used > that to demonstrate that it is indeed picking a mirror it wouldn't > have auto-selected (an auto-select would probably have selected > voxel for me). You can see a list of mirror shortnames to use > here: http://sourceforge.net/apps/trac/sourceforge/wiki/Mirrors > > Also note that this download URL will indeed require a tool that > follows 302 redirects, wget will do that automatically, curl > requires the -L option. I'm not familiar with axel, so I don't know > if that needs any special consideration or not. > > For further reading, we have some info in our community docs on > downloading via command-line here: > http://sourceforge.net/p/forge/community-docs/downloading-via-cli/ > > Regards, Chris Tsai SourceForge Support sfn...@ge... > > > On Mon, Sep 26, 2011 at 2:11 PM, Alexander Hansen > <ale...@gm...> wrote: >> Hi. > >> I'm a developer with the Fink project. A good of the packages we >> maintain are hosted at SourceForge, but we're currently relying >> on redirection of our crufty URLs to find them. :-) We use >> tools like wget, curl, axel, .. to download the source files. > >> The current syntax we're using to pick out individual mirrors >> is, for example, to download directly from: > >> http://voxel.dl.sourceforge.net/sourceforge/ > >> followed by the path to the project's source. These aren't >> resolving anymore and so users have to go through redirects to >> get the sources. > >> Do we have the option to choose a definite mirror, e.g. so our >> users can pick one close to them geographically? > > ==== This e- mail message is intended only for the named > recipient(s) above. It may contain confidential and privileged > information. If you are not the intended recipient you are hereby > notified that any dissemination, distribution or copying of this > e-mail and any attachment(s) is strictly prohibited. If you have > received this e-mail in error, please immediately notify the sender > by replying to this e-mail and delete the message and any > attachment(s) from your system. Thank you. > - -- Alexander Hansen, Ph.D. Fink User Liaison http://finkakh.wordpress.com/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6BBW4ACgkQB8UpO3rKjQ/92ACfTbznIgxau9ic637LTvXiqApQ 8YoAoJ9Sc0AKNASYYWUggKulmPTySxE0 =JGeg -----END PGP SIGNATURE----- |
|
From: Max H. <ma...@qu...> - 2011-09-27 09:57:38
|
Am 27.09.2011 um 01:06 schrieb Alexander Hansen: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 9/26/11 5:04 PM, Alexander Hansen wrote: >> I asked SF.net about how to replace our ailing mirror list with >> something that still contains geographic information and here's >> the response. It looks like we'd need to do some additional coding >> to implement it. > > I've also been playing around with Using > "http://downloads.sourceforge.net/project/<project path>". This works > with wget, but not curl: > [...] > curl --connect-timeout 30 -f -L -A 'fink/0.31.2' -O > http://downloads.sourceforge.net/project/octave/Octave\ Forge\ > Packages/R2009-06-07/multicore-0.2.15.tar.gz > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- > - --:--:-- 0 > curl: (22) The requested URL returned error: 400 > ### execution of curl failed, exit code 22 > Downloading the file "multicore-0.2.15.tar.gz" failed. This error is caused by the spaces in the filename. Wget automatically converts these to %20, but curl doesn't. Once I replaced the spaces manually by %20, curl worked just fine. A simple sed or perl statement can of course automate this. Note that wget will accept the URL with %20, too. So maybe we should URL encode URLs before passing them to the downloader tool? Bye, Max |
|
From: Alexander H. <ale...@gm...> - 2011-09-27 11:41:26
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 9/27/11 5:57 AM, Max Horn wrote: > > Am 27.09.2011 um 01:06 schrieb Alexander Hansen: > >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 >> >> On 9/26/11 5:04 PM, Alexander Hansen wrote: >>> I asked SF.net about how to replace our ailing mirror list with >>> something that still contains geographic information and >>> here's the response. It looks like we'd need to do some >>> additional coding to implement it. >> >> I've also been playing around with Using >> "http://downloads.sourceforge.net/project/<project path>". This >> works with wget, but not curl: >> > [...] > > >> curl --connect-timeout 30 -f -L -A 'fink/0.31.2' -O >> http://downloads.sourceforge.net/project/octave/Octave\ Forge\ >> Packages/R2009-06-07/multicore-0.2.15.tar.gz % Total % >> Received % Xferd Average Speed Time Time Time Current >> Dload Upload Total Spent Left Speed 0 0 0 0 >> 0 0 0 0 --:--:-- --:--:-- - --:--:-- 0 curl: >> (22) The requested URL returned error: 400 ### execution of curl >> failed, exit code 22 Downloading the file >> "multicore-0.2.15.tar.gz" failed. > > This error is caused by the spaces in the filename. Wget > automatically converts these to %20, but curl doesn't. > > Once I replaced the spaces manually by %20, curl worked just fine. > A simple sed or perl statement can of course automate this. Note > that wget will accept the URL with %20, too. So maybe we should URL > encode URLs before passing them to the downloader tool? > > Bye, Max Thanks! I was able to use Source: http://downloads.sourceforge.net/project/octave/Octave%%20Forge%%20Packages... in a .info file and it works for wget, curl, axel, axelautomirror, lftpget, and aria2, so we don't even need to do anything additional for my particular corner case. :-) My thought here is that it looks like we'd have to do significant extra coding to do explicit mirror selection from sourceforge now, so instead we might want to start having packages use Source: http://downloads.sourceforge.net/project/<path/to/tarball/> rather than Source: mirror:sourceforge:<path/to/tarball> and allow sf.net's automatic mirror selector to do its thing. - -- Alexander Hansen, Ph.D. Fink User Liaison http://finkakh.wordpress.com/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6BtlcACgkQB8UpO3rKjQ/6JwCbBAp7pnqN9ZIN5xQNTj8RJsBW ssIAn3oC8q6POD6TEwB2CVr0uryhNanf =HeZR -----END PGP SIGNATURE----- |
|
From: Max H. <ma...@qu...> - 2011-09-27 12:10:12
|
Am 27.09.2011 um 13:41 schrieb Alexander Hansen: [...] > > My thought here is that it looks like we'd have to do significant > extra coding to do explicit mirror selection from sourceforge now, so > instead we might want to start having packages use > > Source: http://downloads.sourceforge.net/project/<path/to/tarball/> > > rather than > > Source: mirror:sourceforge:<path/to/tarball> I agree with the idea that we should not try to fight SF.net's system of automatically choosing a "good" mirror. But do we really need to change packages for this? Can't we just change the definition of the sourceforge mirror to only contain http://downloads.sourceforge.net? Cheers, Max |
|
From: Alexander H. <ale...@gm...> - 2011-09-27 13:07:13
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 9/27/11 8:10 AM, Max Horn wrote: > > Am 27.09.2011 um 13:41 schrieb Alexander Hansen: > > [...] > >> >> My thought here is that it looks like we'd have to do >> significant extra coding to do explicit mirror selection from >> sourceforge now, so instead we might want to start having >> packages use >> >> Source: >> http://downloads.sourceforge.net/project/<path/to/tarball/> >> >> rather than >> >> Source: mirror:sourceforge:<path/to/tarball> > > I agree with the idea that we should not try to fight SF.net's > system of automatically choosing a "good" mirror. But do we really > need to change packages for this? Can't we just change the > definition of the sourceforge mirror to only contain > http://downloads.sourceforge.net? > > Cheers, Max Sounds good to me. I've verified that http://downloads.sourceforge.net/project/ works as a mirror:sourceforge entry. If nobody has objections, I can set this up in HEAD. One thing I've noted is that sourceforge's automatic mirror chooser sometimes will pick a mirror that doesn't work, so users would need to retry the "same" URL--which should get kicked over to a new mirror. - -- Alexander Hansen, Ph.D. Fink User Liaison http://finkakh.wordpress.com/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6BynYACgkQB8UpO3rKjQ+aywCfefmA8bM0IL4gHHQCHIzD/9+j /AkAnAoMdiifjwQhk9nL2YBBLo7SHxUP =cYTf -----END PGP SIGNATURE----- |
