Re: [Jython-dev] unicode test failures urllib tests etc.

[Jeff A. <ja...@fa...>]

The tests use a system of "resources" to switch on or off tests. There 
is a "network" resource that if not enabled (regrtest -u) is intended to 
switch off those tests that make external reference and also a 
"urlfetch" resource, and this is what test_normalization uses. There is 
a test_support.requires_resource decorator applicable test by test.

If Travis blocks external reference, then we ought to run tests there 
without that resource so we don't try. Or is that the rate at which we 
launch them just too high? We could consider defining another, as long 
as we're clear what we mean and document it. But using "urlfetch" in a 
few more places may express what we really mean: existing is always 
better than bespoke.

On the substantive issue, we decided not to support lone surrogates in 
Jython and then changed our minds to allow the (ab)use by which CPython 
smuggle bytes in file paths that exist apart from their intended 
encoding. Some design is based on the former state. I would say that 
when a lone surrogate is encountered it will still depend on the context 
what it means. (ISTR it's only 0xDC00..0xDCFF that's at issue.)

Jeff

Jeff Allen

On 02/11/2018 03:32, Adam Burke wrote:
> Interesting. Apart from the unintentional DOS, I feel unit / regr test 
> time dependencies on remote resources should be limited anyway.
>
> How are you thinking of resolving it? One way would be to keep a local 
> copy of the relevant file, and declare the dependency similarly to a 
> library, perhaps in the gradle build files. The metadata that this is 
> a copy of someone else’s golden source seems valuable.
>
> Cheers
> Adam
>
> 在 2018年10月26日，上午4:20，Raymond Ferguson via Jython-dev 
> <jyt...@li... 
> <mailto:jyt...@li...>> 写道：
>
>>  Finally managed to get some verbose output.
>>
>> ./Lib/test/test_urllib2net.py:   FTP_HOST = 
>> "ftp://ftp.mirror.nl/pub/gnu/"
>>
>> We're probably participating in an unintentional DOS attack via 
>> github->travis and just being blocked.
>>
>>      [exec] Re-running failed tests in verbose mode
>>      [exec] Re-running test 'test_urllib2net' in verbose mode
>>      [exec] test test_urllib2net failed -- Traceback (most recent 
>> call last):
>>      [exec]   File 
>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", 
>> line 312, in test_ftp_no_timeout
>>      [exec]     u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
>>      [exec]   File 
>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", 
>> line 27, in wrapped
>>      [exec]  return _retry_thrice(func, exc, *args, **kwargs)
>>      [exec]   File 
>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", 
>> line 23, in _retry_thrice
>>      [exec]     raise last_exc
>>      [exec] URLError: <urlopen error ftp error: 425 Security: Bad IP 
>> connecting.>
>> *Ray Ferguson*
>> *Enterprise Solutions Developer*
>> *Devendor Tech LLC*
>> *ray...@de...* <mailto:ray...@de...>
>> *W: 608-960-8090C: 608-516-9391*
>>
>>
>> ᐧ
>>
>> On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson 
>> <ray...@de... 
>> <mailto:ray...@de...>> wrote:
>>
>>     There is this little bit of code in __builtin__.unichr that
>>     creates a hole in unicode values for utf-16 encoding.
>>
>>      if (i >= 0xD800 && i <= 0xDFFF) {
>>                 throw Py.ValueError("unichr() arg is a lone surrogate
>>     in range (0xD800, 0xDFFF) (Jython UTF-16 encoding)");
>>     }
>>
>>     Anyway... I don't see any way that test_normalization can pass
>>     given that sys.max_unicode > 0xd800 and one of the tests does a
>>     for range(sys.maxunicode) unichr(...).
>>
>>     Strangely though, this does pass from travis-ci.org
>>     <http://travis-ci.org>. Apparently there is some sort of reactive
>>     blocking in place that will drop connections to www.unicode.org
>>     <http://www.unicode.org> from your IP if you trigger something,
>>     probably rate of get requests for test files.  The test in
>>     question grabs a file from www.unicode.org
>>     <http://www.unicode.org> and skips the test on timeout, so it
>>     doesn't hit the part of the test that will always fail.
>>
>>     Anyway, after troubleshooting for a while, I managed to trigger
>>     this and suddenly can't reach unicode.org <http://unicode.org>
>>     from my LAN, but if I switch my phone over to mobile data for a
>>     new source address, clearly the site is still up.  I can even
>>     ping it but tcp syn is dropped.
>>
>>     I've also noticed that urllib2 tests work fine from my network,
>>     but fail on travis-ci.  My guess is that the at least one of the
>>     test targets are taking steps to reduce impact of being hard
>>     coded into so many CI tests.
>>
>>     *Ray Ferguson*
>>     *Enterprise Solutions Developer*
>>     *Devendor Tech LLC*
>>     *ray...@de...*
>>     <mailto:ray...@de...>
>>     *W: 608-960-8090C: 608-516-9391*
>>