Re: [Jython-dev] unicode test failures urllib tests etc.

[Adam B. <ada...@gm...>]

The resource system in the regrtests is sensible. But it seems a shame to
make the network resource a prerequisite for what is essentially a static,
versioned file?

Cheers
Adam

On Fri, 2 Nov 2018 at 17:42, Jeff Allen <ja...@fa...> wrote:

> The tests use a system of "resources" to switch on or off tests. There is
> a "network" resource that if not enabled (regrtest -u) is intended to
> switch off those tests that make external reference and also a "urlfetch"
> resource, and this is what test_normalization uses. There is a
> test_support.requires_resource decorator applicable test by test.
>
> If Travis blocks external reference, then we ought to run tests there
> without that resource so we don't try. Or is that the rate at which we
> launch them just too high? We could consider defining another, as long as
> we're clear what we mean and document it. But using "urlfetch" in a few
> more places may express what we really mean: existing is always better than
> bespoke.
> On the substantive issue, we decided not to support lone surrogates in
> Jython and then changed our minds to allow the (ab)use by which CPython
> smuggle bytes in file paths that exist apart from their intended encoding.
> Some design is based on the former state. I would say that when a lone
> surrogate is encountered it will still depend on the context what it means.
> (ISTR it's only 0xDC00..0xDCFF that's at issue.)
>
> Jeff
>
> Jeff Allen
>
> On 02/11/2018 03:32, Adam Burke wrote:
>
> Interesting. Apart from the unintentional DOS, I feel unit / regr test
> time dependencies on remote resources should be limited anyway.
>
> How are you thinking of resolving it? One way would be to keep a local
> copy of the relevant file, and declare the dependency similarly to a
> library, perhaps in the gradle build files. The metadata that this is a
> copy of someone else’s golden source seems valuable.
>
> Cheers
> Adam
>
> 在 2018年10月26日，上午4:20，Raymond Ferguson via Jython-dev <
> jyt...@li...> 写道：
>
>  Finally managed to get some verbose output.
>
> ./Lib/test/test_urllib2net.py:    FTP_HOST = "ftp://ftp.mirror.nl/pub/gnu/
> "
>
> We're probably participating in an unintentional DOS attack via
> github->travis and just being blocked.
>
>      [exec] Re-running failed tests in verbose mode
>      [exec] Re-running test 'test_urllib2net' in verbose mode
>      [exec] test test_urllib2net failed -- Traceback (most recent call
> last):
>      [exec]   File
> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
> 312, in test_ftp_no_timeout
>      [exec]     u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
>      [exec]   File
> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
> 27, in wrapped
>      [exec]     return _retry_thrice(func, exc, *args, **kwargs)
>      [exec]   File
> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
> 23, in _retry_thrice
>      [exec]     raise last_exc
>      [exec] URLError: <urlopen error ftp error: 425 Security: Bad IP
> connecting.>
> *Ray Ferguson*
> *Enterprise Solutions Developer*
> *Devendor Tech LLC*
> *ray...@de...* <ray...@de...>
> *W: 608-960-8090 C: 608-516-9391*
>
>
> ᐧ
>
> On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson <
> ray...@de...> wrote:
>
>> There is this little bit of code in __builtin__.unichr that creates a
>> hole in unicode values for utf-16 encoding.
>>
>>  if (i >= 0xD800 && i <= 0xDFFF) {
>>             throw Py.ValueError("unichr() arg is a lone surrogate in
>> range (0xD800, 0xDFFF) (Jython UTF-16 encoding)");
>> }
>>
>> Anyway... I don't see any way that test_normalization can pass given that
>> sys.max_unicode > 0xd800 and one of the tests does a for
>> range(sys.maxunicode) unichr(...).
>>
>> Strangely though, this does pass from travis-ci.org.  Apparently there
>> is some sort of reactive blocking in place that will drop connections to
>> www.unicode.org from your IP if you trigger something, probably rate of
>> get requests for test files.  The test in question grabs a file from
>> www.unicode.org and skips the test on timeout, so it doesn't hit the
>> part of the test that will always fail.
>>
>> Anyway, after troubleshooting for a while, I managed to trigger this and
>> suddenly can't reach unicode.org from my LAN, but if I switch my phone
>> over to mobile data for a new source address, clearly the site is still
>> up.  I can even ping it but tcp syn is dropped.
>>
>> I've also noticed that urllib2 tests work fine from my network, but fail
>> on travis-ci.  My guess is that the at least one of the test targets are
>> taking steps to reduce impact of being hard coded into so many CI tests.
>>
>> *Ray Ferguson*
>> *Enterprise Solutions Developer*
>> *Devendor Tech LLC*
>> *ray...@de...* <ray...@de...>
>> *W: 608-960-8090 C: 608-516-9391*
>>
>>
>