Menu
▾
▴
Re: [Jython-dev] unicode test failures urllib tests etc.
|
From: Jeff A. <ja...@fa...> - 2018-11-02 08:38:13
|
It's 2.4MB, which is big but not huge, but then there is an automatic adaptation to version in test_normalization, and we'd have to remember to do that. Also, it becomes another obstacle to reverting test_normalization to the stdlib version, which is a standing objective for every test. But this is solving the wrong problem. The stack dump Ray exhibits is from a different test, at a different site, where the access is not guarded by "urlfetch". In fact I think we're talking about rejection by the FTP server itself, for which accessing a file is not a substitute test. Jeff Allen On 02/11/2018 07:48, Adam Burke wrote: > The resource system in the regrtests is sensible. But it seems a shame > to make the network resource a prerequisite for what is essentially a > static, versioned file? > > Cheers > Adam > > On Fri, 2 Nov 2018 at 17:42, Jeff Allen <ja...@fa... > <mailto:ja...@fa...>> wrote: > > The tests use a system of "resources" to switch on or off tests. > There is a "network" resource that if not enabled (regrtest -u) is > intended to switch off those tests that make external reference > and also a "urlfetch" resource, and this is what > test_normalization uses. There is a test_support.requires_resource > decorator applicable test by test. > > If Travis blocks external reference, then we ought to run tests > there without that resource so we don't try. Or is that the rate > at which we launch them just too high? We could consider defining > another, as long as we're clear what we mean and document it. But > using "urlfetch" in a few more places may express what we really > mean: existing is always better than bespoke. > > On the substantive issue, we decided not to support lone > surrogates in Jython and then changed our minds to allow the > (ab)use by which CPython smuggle bytes in file paths that exist > apart from their intended encoding. Some design is based on the > former state. I would say that when a lone surrogate is > encountered it will still depend on the context what it means. > (ISTR it's only 0xDC00..0xDCFF that's at issue.) > > Jeff > > Jeff Allen > > On 02/11/2018 03:32, Adam Burke wrote: >> Interesting. Apart from the unintentional DOS, I feel unit / regr >> test time dependencies on remote resources should be limited anyway. >> >> How are you thinking of resolving it? One way would be to keep a >> local copy of the relevant file, and declare the dependency >> similarly to a library, perhaps in the gradle build files. The >> metadata that this is a copy of someone else’s golden source >> seems valuable. >> >> Cheers >> Adam >> >> 在 2018年10月26日，上午4:20，Raymond Ferguson via Jython-dev >> <jyt...@li... >> <mailto:jyt...@li...>> 写道： >> >>> Finally managed to get some verbose output. >>> >>> ./Lib/test/test_urllib2net.py: FTP_HOST = >>> "ftp://ftp.mirror.nl/pub/gnu/" >>> >>> We're probably participating in an unintentional DOS attack via >>> github->travis and just being blocked. >>> >>> [exec] Re-running failed tests in verbose mode >>> [exec] Re-running test 'test_urllib2net' in verbose mode >>> [exec] test test_urllib2net failed -- Traceback (most >>> recent call last): >>> [exec] File >>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", >>> line 312, in test_ftp_no_timeout >>> [exec] u = _urlopen_with_retry(self.FTP_HOST, timeout=None) >>> [exec] File >>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", >>> line 27, in wrapped >>> [exec] return _retry_thrice(func, exc, *args, **kwargs) >>> [exec] File >>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", >>> line 23, in _retry_thrice >>> [exec] raise last_exc >>> [exec] URLError: <urlopen error ftp error: 425 Security: >>> Bad IP connecting.> >>> *Ray Ferguson* >>> *Enterprise Solutions Developer* >>> *Devendor Tech LLC* >>> *ray...@de...* >>> <mailto:ray...@de...> >>> *W: 608-960-8090C: 608-516-9391* >>> >>> >>> ᐧ >>> >>> On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson >>> <ray...@de... >>> <mailto:ray...@de...>> wrote: >>> >>> There is this little bit of code in __builtin__.unichr that >>> creates a hole in unicode values for utf-16 encoding. >>> >>> if (i >= 0xD800 && i <= 0xDFFF) { >>> throw Py.ValueError("unichr() arg is a lone surrogate in >>> range (0xD800, 0xDFFF) (Jython UTF-16 encoding)"); >>> } >>> >>> Anyway... I don't see any way that test_normalization can >>> pass given that sys.max_unicode > 0xd800 and one of the >>> tests does a for range(sys.maxunicode) unichr(...). >>> >>> Strangely though, this does pass from travis-ci.org >>> <http://travis-ci.org>. Apparently there is some sort of >>> reactive blocking in place that will drop connections to >>> www.unicode.org <http://www.unicode.org> from your IP if you >>> trigger something, probably rate of get requests for test >>> files. The test in question grabs a file from >>> www.unicode.org <http://www.unicode.org> and skips the test >>> on timeout, so it doesn't hit the part of the test that will >>> always fail. >>> >>> Anyway, after troubleshooting for a while, I managed to >>> trigger this and suddenly can't reach unicode.org >>> <http://unicode.org> from my LAN, but if I switch my phone >>> over to mobile data for a new source address, clearly the >>> site is still up. I can even ping it but tcp syn is dropped. >>> >>> I've also noticed that urllib2 tests work fine from my >>> network, but fail on travis-ci. My guess is that the at >>> least one of the test targets are taking steps to reduce >>> impact of being hard coded into so many CI tests. >>> >>> *Ray Ferguson* >>> *Enterprise Solutions Developer* >>> *Devendor Tech LLC* >>> *ray...@de...* >>> <mailto:ray...@de...> >>> *W: 608-960-8090C: 608-516-9391* >>> > |
