jython-dev

[Jython-dev] unicode test failures urllib tests etc.

[Raymond F. <ray...@de...>]

There is this little bit of code in __builtin__.unichr that creates a hole
in unicode values for utf-16 encoding.

 if (i >= 0xD800 && i <= 0xDFFF) {
            throw Py.ValueError("unichr() arg is a lone surrogate in range
(0xD800, 0xDFFF) (Jython UTF-16 encoding)");
}

Anyway... I don't see any way that test_normalization can pass given that
sys.max_unicode > 0xd800 and one of the tests does a for
range(sys.maxunicode) unichr(...).

Strangely though, this does pass from travis-ci.org.  Apparently there is
some sort of reactive blocking in place that will drop connections to
www.unicode.org from your IP if you trigger something, probably rate of get
requests for test files.  The test in question grabs a file from
www.unicode.org and skips the test on timeout, so it doesn't hit the part
of the test that will always fail.

Anyway, after troubleshooting for a while, I managed to trigger this and
suddenly can't reach unicode.org from my LAN, but if I switch my phone over
to mobile data for a new source address, clearly the site is still up.  I
can even ping it but tcp syn is dropped.

I've also noticed that urllib2 tests work fine from my network, but fail on
travis-ci.  My guess is that the at least one of the test targets are
taking steps to reduce impact of being hard coded into so many CI tests.

*Ray Ferguson*
*Enterprise Solutions Developer*
*Devendor Tech LLC*
*ray...@de...* <ray...@de...>
*W: 608-960-8090 C: 608-516-9391*

ᐧ

Re: [Jython-dev] unicode test failures urllib tests etc.

[Raymond F. <ray...@de...>]

 Finally managed to get some verbose output.

./Lib/test/test_urllib2net.py:    FTP_HOST = "ftp://ftp.mirror.nl/pub/gnu/"

We're probably participating in an unintentional DOS attack via
github->travis and just being blocked.

     [exec] Re-running failed tests in verbose mode
     [exec] Re-running test 'test_urllib2net' in verbose mode
     [exec] test test_urllib2net failed -- Traceback (most recent call
last):
     [exec]   File
"/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
312, in test_ftp_no_timeout
     [exec]     u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
     [exec]   File
"/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
27, in wrapped
     [exec]     return _retry_thrice(func, exc, *args, **kwargs)
     [exec]   File
"/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
23, in _retry_thrice
     [exec]     raise last_exc
     [exec] URLError: <urlopen error ftp error: 425 Security: Bad IP
connecting.>
*Ray Ferguson*
*Enterprise Solutions Developer*
*Devendor Tech LLC*
*ray...@de...* <ray...@de...>
*W: 608-960-8090 C: 608-516-9391*


ᐧ

On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson <
ray...@de...> wrote:

> There is this little bit of code in __builtin__.unichr that creates a hole
> in unicode values for utf-16 encoding.
>
>  if (i >= 0xD800 && i <= 0xDFFF) {
>             throw Py.ValueError("unichr() arg is a lone surrogate in range
> (0xD800, 0xDFFF) (Jython UTF-16 encoding)");
> }
>
> Anyway... I don't see any way that test_normalization can pass given that
> sys.max_unicode > 0xd800 and one of the tests does a for
> range(sys.maxunicode) unichr(...).
>
> Strangely though, this does pass from travis-ci.org.  Apparently there is
> some sort of reactive blocking in place that will drop connections to
> www.unicode.org from your IP if you trigger something, probably rate of
> get requests for test files.  The test in question grabs a file from
> www.unicode.org and skips the test on timeout, so it doesn't hit the part
> of the test that will always fail.
>
> Anyway, after troubleshooting for a while, I managed to trigger this and
> suddenly can't reach unicode.org from my LAN, but if I switch my phone
> over to mobile data for a new source address, clearly the site is still
> up.  I can even ping it but tcp syn is dropped.
>
> I've also noticed that urllib2 tests work fine from my network, but fail
> on travis-ci.  My guess is that the at least one of the test targets are
> taking steps to reduce impact of being hard coded into so many CI tests.
>
> *Ray Ferguson*
> *Enterprise Solutions Developer*
> *Devendor Tech LLC*
> *ray...@de...* <ray...@de...>
> *W: 608-960-8090 C: 608-516-9391*
>
> ᐧ
>

Re: [Jython-dev] unicode test failures urllib tests etc.

[Adam B. <ada...@gm...>]

Interesting. Apart from the unintentional DOS, I feel unit / regr test time
dependencies on remote resources should be limited anyway.

How are you thinking of resolving it? One way would be to keep a local copy
of the relevant file, and declare the dependency similarly to a library,
perhaps in the gradle build files. The metadata that this is a copy of
someone else’s golden source seems valuable.

Cheers
Adam

在 2018年10月26日，上午4:20，Raymond Ferguson via Jython-dev <
jyt...@li...> 写道：

 Finally managed to get some verbose output.

./Lib/test/test_urllib2net.py:    FTP_HOST = "ftp://ftp.mirror.nl/pub/gnu/"

We're probably participating in an unintentional DOS attack via
github->travis and just being blocked.

     [exec] Re-running failed tests in verbose mode
     [exec] Re-running test 'test_urllib2net' in verbose mode
     [exec] test test_urllib2net failed -- Traceback (most recent call
last):
     [exec]   File
"/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
312, in test_ftp_no_timeout
     [exec]     u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
     [exec]   File
"/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
27, in wrapped
     [exec]     return _retry_thrice(func, exc, *args, **kwargs)
     [exec]   File
"/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
23, in _retry_thrice
     [exec]     raise last_exc
     [exec] URLError: <urlopen error ftp error: 425 Security: Bad IP
connecting.>
*Ray Ferguson*
*Enterprise Solutions Developer*
*Devendor Tech LLC*
*ray...@de...* <ray...@de...>
*W: 608-960-8090 C: 608-516-9391*


ᐧ

On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson <
ray...@de...> wrote:

> There is this little bit of code in __builtin__.unichr that creates a hole
> in unicode values for utf-16 encoding.
>
>  if (i >= 0xD800 && i <= 0xDFFF) {
>             throw Py.ValueError("unichr() arg is a lone surrogate in range
> (0xD800, 0xDFFF) (Jython UTF-16 encoding)");
> }
>
> Anyway... I don't see any way that test_normalization can pass given that
> sys.max_unicode > 0xd800 and one of the tests does a for
> range(sys.maxunicode) unichr(...).
>
> Strangely though, this does pass from travis-ci.org.  Apparently there is
> some sort of reactive blocking in place that will drop connections to
> www.unicode.org from your IP if you trigger something, probably rate of
> get requests for test files.  The test in question grabs a file from
> www.unicode.org and skips the test on timeout, so it doesn't hit the part
> of the test that will always fail.
>
> Anyway, after troubleshooting for a while, I managed to trigger this and
> suddenly can't reach unicode.org from my LAN, but if I switch my phone
> over to mobile data for a new source address, clearly the site is still
> up.  I can even ping it but tcp syn is dropped.
>
> I've also noticed that urllib2 tests work fine from my network, but fail
> on travis-ci.  My guess is that the at least one of the test targets are
> taking steps to reduce impact of being hard coded into so many CI tests.
>
> *Ray Ferguson*
> *Enterprise Solutions Developer*
> *Devendor Tech LLC*
> *ray...@de...* <ray...@de...>
> *W: 608-960-8090 C: 608-516-9391*
>
> ᐧ
>
_______________________________________________
Jython-dev mailing list
Jyt...@li...
https://lists.sourceforge.net/lists/listinfo/jython-dev

Re: [Jython-dev] unicode test failures urllib tests etc.

[Jeff A. <ja...@fa...>]

The tests use a system of "resources" to switch on or off tests. There 
is a "network" resource that if not enabled (regrtest -u) is intended to 
switch off those tests that make external reference and also a 
"urlfetch" resource, and this is what test_normalization uses. There is 
a test_support.requires_resource decorator applicable test by test.

If Travis blocks external reference, then we ought to run tests there 
without that resource so we don't try. Or is that the rate at which we 
launch them just too high? We could consider defining another, as long 
as we're clear what we mean and document it. But using "urlfetch" in a 
few more places may express what we really mean: existing is always 
better than bespoke.

On the substantive issue, we decided not to support lone surrogates in 
Jython and then changed our minds to allow the (ab)use by which CPython 
smuggle bytes in file paths that exist apart from their intended 
encoding. Some design is based on the former state. I would say that 
when a lone surrogate is encountered it will still depend on the context 
what it means. (ISTR it's only 0xDC00..0xDCFF that's at issue.)

Jeff

Jeff Allen

On 02/11/2018 03:32, Adam Burke wrote:
> Interesting. Apart from the unintentional DOS, I feel unit / regr test 
> time dependencies on remote resources should be limited anyway.
>
> How are you thinking of resolving it? One way would be to keep a local 
> copy of the relevant file, and declare the dependency similarly to a 
> library, perhaps in the gradle build files. The metadata that this is 
> a copy of someone else’s golden source seems valuable.
>
> Cheers
> Adam
>
> 在 2018年10月26日，上午4:20，Raymond Ferguson via Jython-dev 
> <jyt...@li... 
> <mailto:jyt...@li...>> 写道：
>
>>  Finally managed to get some verbose output.
>>
>> ./Lib/test/test_urllib2net.py:   FTP_HOST = 
>> "ftp://ftp.mirror.nl/pub/gnu/"
>>
>> We're probably participating in an unintentional DOS attack via 
>> github->travis and just being blocked.
>>
>>      [exec] Re-running failed tests in verbose mode
>>      [exec] Re-running test 'test_urllib2net' in verbose mode
>>      [exec] test test_urllib2net failed -- Traceback (most recent 
>> call last):
>>      [exec]   File 
>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", 
>> line 312, in test_ftp_no_timeout
>>      [exec]     u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
>>      [exec]   File 
>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", 
>> line 27, in wrapped
>>      [exec]  return _retry_thrice(func, exc, *args, **kwargs)
>>      [exec]   File 
>> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", 
>> line 23, in _retry_thrice
>>      [exec]     raise last_exc
>>      [exec] URLError: <urlopen error ftp error: 425 Security: Bad IP 
>> connecting.>
>> *Ray Ferguson*
>> *Enterprise Solutions Developer*
>> *Devendor Tech LLC*
>> *ray...@de...* <mailto:ray...@de...>
>> *W: 608-960-8090C: 608-516-9391*
>>
>>
>> ᐧ
>>
>> On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson 
>> <ray...@de... 
>> <mailto:ray...@de...>> wrote:
>>
>>     There is this little bit of code in __builtin__.unichr that
>>     creates a hole in unicode values for utf-16 encoding.
>>
>>      if (i >= 0xD800 && i <= 0xDFFF) {
>>                 throw Py.ValueError("unichr() arg is a lone surrogate
>>     in range (0xD800, 0xDFFF) (Jython UTF-16 encoding)");
>>     }
>>
>>     Anyway... I don't see any way that test_normalization can pass
>>     given that sys.max_unicode > 0xd800 and one of the tests does a
>>     for range(sys.maxunicode) unichr(...).
>>
>>     Strangely though, this does pass from travis-ci.org
>>     <http://travis-ci.org>. Apparently there is some sort of reactive
>>     blocking in place that will drop connections to www.unicode.org
>>     <http://www.unicode.org> from your IP if you trigger something,
>>     probably rate of get requests for test files.  The test in
>>     question grabs a file from www.unicode.org
>>     <http://www.unicode.org> and skips the test on timeout, so it
>>     doesn't hit the part of the test that will always fail.
>>
>>     Anyway, after troubleshooting for a while, I managed to trigger
>>     this and suddenly can't reach unicode.org <http://unicode.org>
>>     from my LAN, but if I switch my phone over to mobile data for a
>>     new source address, clearly the site is still up.  I can even
>>     ping it but tcp syn is dropped.
>>
>>     I've also noticed that urllib2 tests work fine from my network,
>>     but fail on travis-ci.  My guess is that the at least one of the
>>     test targets are taking steps to reduce impact of being hard
>>     coded into so many CI tests.
>>
>>     *Ray Ferguson*
>>     *Enterprise Solutions Developer*
>>     *Devendor Tech LLC*
>>     *ray...@de...*
>>     <mailto:ray...@de...>
>>     *W: 608-960-8090C: 608-516-9391*
>>

Re: [Jython-dev] unicode test failures urllib tests etc.

[Adam B. <ada...@gm...>]

The resource system in the regrtests is sensible. But it seems a shame to
make the network resource a prerequisite for what is essentially a static,
versioned file?

Cheers
Adam

On Fri, 2 Nov 2018 at 17:42, Jeff Allen <ja...@fa...> wrote:

> The tests use a system of "resources" to switch on or off tests. There is
> a "network" resource that if not enabled (regrtest -u) is intended to
> switch off those tests that make external reference and also a "urlfetch"
> resource, and this is what test_normalization uses. There is a
> test_support.requires_resource decorator applicable test by test.
>
> If Travis blocks external reference, then we ought to run tests there
> without that resource so we don't try. Or is that the rate at which we
> launch them just too high? We could consider defining another, as long as
> we're clear what we mean and document it. But using "urlfetch" in a few
> more places may express what we really mean: existing is always better than
> bespoke.
> On the substantive issue, we decided not to support lone surrogates in
> Jython and then changed our minds to allow the (ab)use by which CPython
> smuggle bytes in file paths that exist apart from their intended encoding.
> Some design is based on the former state. I would say that when a lone
> surrogate is encountered it will still depend on the context what it means.
> (ISTR it's only 0xDC00..0xDCFF that's at issue.)
>
> Jeff
>
> Jeff Allen
>
> On 02/11/2018 03:32, Adam Burke wrote:
>
> Interesting. Apart from the unintentional DOS, I feel unit / regr test
> time dependencies on remote resources should be limited anyway.
>
> How are you thinking of resolving it? One way would be to keep a local
> copy of the relevant file, and declare the dependency similarly to a
> library, perhaps in the gradle build files. The metadata that this is a
> copy of someone else’s golden source seems valuable.
>
> Cheers
> Adam
>
> 在 2018年10月26日，上午4:20，Raymond Ferguson via Jython-dev <
> jyt...@li...> 写道：
>
>  Finally managed to get some verbose output.
>
> ./Lib/test/test_urllib2net.py:    FTP_HOST = "ftp://ftp.mirror.nl/pub/gnu/
> "
>
> We're probably participating in an unintentional DOS attack via
> github->travis and just being blocked.
>
>      [exec] Re-running failed tests in verbose mode
>      [exec] Re-running test 'test_urllib2net' in verbose mode
>      [exec] test test_urllib2net failed -- Traceback (most recent call
> last):
>      [exec]   File
> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
> 312, in test_ftp_no_timeout
>      [exec]     u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
>      [exec]   File
> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
> 27, in wrapped
>      [exec]     return _retry_thrice(func, exc, *args, **kwargs)
>      [exec]   File
> "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py", line
> 23, in _retry_thrice
>      [exec]     raise last_exc
>      [exec] URLError: <urlopen error ftp error: 425 Security: Bad IP
> connecting.>
> *Ray Ferguson*
> *Enterprise Solutions Developer*
> *Devendor Tech LLC*
> *ray...@de...* <ray...@de...>
> *W: 608-960-8090 C: 608-516-9391*
>
>
> ᐧ
>
> On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson <
> ray...@de...> wrote:
>
>> There is this little bit of code in __builtin__.unichr that creates a
>> hole in unicode values for utf-16 encoding.
>>
>>  if (i >= 0xD800 && i <= 0xDFFF) {
>>             throw Py.ValueError("unichr() arg is a lone surrogate in
>> range (0xD800, 0xDFFF) (Jython UTF-16 encoding)");
>> }
>>
>> Anyway... I don't see any way that test_normalization can pass given that
>> sys.max_unicode > 0xd800 and one of the tests does a for
>> range(sys.maxunicode) unichr(...).
>>
>> Strangely though, this does pass from travis-ci.org.  Apparently there
>> is some sort of reactive blocking in place that will drop connections to
>> www.unicode.org from your IP if you trigger something, probably rate of
>> get requests for test files.  The test in question grabs a file from
>> www.unicode.org and skips the test on timeout, so it doesn't hit the
>> part of the test that will always fail.
>>
>> Anyway, after troubleshooting for a while, I managed to trigger this and
>> suddenly can't reach unicode.org from my LAN, but if I switch my phone
>> over to mobile data for a new source address, clearly the site is still
>> up.  I can even ping it but tcp syn is dropped.
>>
>> I've also noticed that urllib2 tests work fine from my network, but fail
>> on travis-ci.  My guess is that the at least one of the test targets are
>> taking steps to reduce impact of being hard coded into so many CI tests.
>>
>> *Ray Ferguson*
>> *Enterprise Solutions Developer*
>> *Devendor Tech LLC*
>> *ray...@de...* <ray...@de...>
>> *W: 608-960-8090 C: 608-516-9391*
>>
>>
>

Re: [Jython-dev] unicode test failures urllib tests etc.

[Jeff A. <ja...@fa...>]

It's 2.4MB, which is big but not huge, but then there is an automatic 
adaptation to version in test_normalization, and we'd have to remember 
to do that. Also, it becomes another obstacle to reverting 
test_normalization to the stdlib version, which is a standing objective 
for every test.

But this is solving the wrong problem.

The stack dump Ray exhibits is from a different test, at a different 
site, where the access is not guarded by "urlfetch". In fact I think 
we're talking about rejection by the FTP server itself, for which 
accessing a file is not a substitute test.

Jeff Allen

On 02/11/2018 07:48, Adam Burke wrote:
> The resource system in the regrtests is sensible. But it seems a shame 
> to make the network resource a prerequisite for what is essentially a 
> static, versioned file?
>
> Cheers
> Adam
>
> On Fri, 2 Nov 2018 at 17:42, Jeff Allen <ja...@fa... 
> <mailto:ja...@fa...>> wrote:
>
>     The tests use a system of "resources" to switch on or off tests.
>     There is a "network" resource that if not enabled (regrtest -u) is
>     intended to switch off those tests that make external reference
>     and also a "urlfetch" resource, and this is what
>     test_normalization uses. There is a test_support.requires_resource
>     decorator applicable test by test.
>
>     If Travis blocks external reference, then we ought to run tests
>     there without that resource so we don't try. Or is that the rate
>     at which we launch them just too high? We could consider defining
>     another, as long as we're clear what we mean and document it. But
>     using "urlfetch" in a few more places may express what we really
>     mean: existing is always better than bespoke.
>
>     On the substantive issue, we decided not to support lone
>     surrogates in Jython and then changed our minds to allow the
>     (ab)use by which CPython smuggle bytes in file paths that exist
>     apart from their intended encoding. Some design is based on the
>     former state. I would say that when a lone surrogate is
>     encountered it will still depend on the context what it means.
>     (ISTR it's only 0xDC00..0xDCFF that's at issue.)
>
>     Jeff
>
>     Jeff Allen
>
>     On 02/11/2018 03:32, Adam Burke wrote:
>>     Interesting. Apart from the unintentional DOS, I feel unit / regr
>>     test time dependencies on remote resources should be limited anyway.
>>
>>     How are you thinking of resolving it? One way would be to keep a
>>     local copy of the relevant file, and declare the dependency
>>     similarly to a library, perhaps in the gradle build files. The
>>     metadata that this is a copy of someone else’s golden source
>>     seems valuable.
>>
>>     Cheers
>>     Adam
>>
>>     在 2018年10月26日，上午4:20，Raymond Ferguson via Jython-dev
>>     <jyt...@li...
>>     <mailto:jyt...@li...>> 写道：
>>
>>>      Finally managed to get some verbose output.
>>>
>>>     ./Lib/test/test_urllib2net.py:   FTP_HOST =
>>>     "ftp://ftp.mirror.nl/pub/gnu/"
>>>
>>>     We're probably participating in an unintentional DOS attack via
>>>     github->travis and just being blocked.
>>>
>>>          [exec] Re-running failed tests in verbose mode
>>>          [exec] Re-running test 'test_urllib2net' in verbose mode
>>>          [exec] test test_urllib2net failed -- Traceback (most
>>>     recent call last):
>>>          [exec]  File
>>>     "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py",
>>>     line 312, in test_ftp_no_timeout
>>>          [exec]    u = _urlopen_with_retry(self.FTP_HOST, timeout=None)
>>>          [exec]  File
>>>     "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py",
>>>     line 27, in wrapped
>>>          [exec]    return _retry_thrice(func, exc, *args, **kwargs)
>>>          [exec]  File
>>>     "/home/travis/build/devendor/jython/dist/Lib/test/test_urllib2net.py",
>>>     line 23, in _retry_thrice
>>>          [exec]    raise last_exc
>>>          [exec] URLError: <urlopen error ftp error: 425 Security:
>>>     Bad IP connecting.>
>>>     *Ray Ferguson*
>>>     *Enterprise Solutions Developer*
>>>     *Devendor Tech LLC*
>>>     *ray...@de...*
>>>     <mailto:ray...@de...>
>>>     *W: 608-960-8090C: 608-516-9391*
>>>
>>>
>>>     ᐧ
>>>
>>>     On Thu, Oct 25, 2018 at 1:14 PM Raymond Ferguson
>>>     <ray...@de...
>>>     <mailto:ray...@de...>> wrote:
>>>
>>>         There is this little bit of code in __builtin__.unichr that
>>>         creates a hole in unicode values for utf-16 encoding.
>>>
>>>          if (i >= 0xD800 && i <= 0xDFFF) {
>>>           throw Py.ValueError("unichr() arg is a lone surrogate in
>>>         range (0xD800, 0xDFFF) (Jython UTF-16 encoding)");
>>>         }
>>>
>>>         Anyway... I don't see any way that test_normalization can
>>>         pass given that sys.max_unicode > 0xd800 and one of the
>>>         tests does a for range(sys.maxunicode) unichr(...).
>>>
>>>         Strangely though, this does pass from travis-ci.org
>>>         <http://travis-ci.org>. Apparently there is some sort of
>>>         reactive blocking in place that will drop connections to
>>>         www.unicode.org <http://www.unicode.org> from your IP if you
>>>         trigger something, probably rate of get requests for test
>>>         files.  The test in question grabs a file from
>>>         www.unicode.org <http://www.unicode.org> and skips the test
>>>         on timeout, so it doesn't hit the part of the test that will
>>>         always fail.
>>>
>>>         Anyway, after troubleshooting for a while, I managed to
>>>         trigger this and suddenly can't reach unicode.org
>>>         <http://unicode.org> from my LAN, but if I switch my phone
>>>         over to mobile data for a new source address, clearly the
>>>         site is still up.  I can even ping it but tcp syn is dropped.
>>>
>>>         I've also noticed that urllib2 tests work fine from my
>>>         network, but fail on travis-ci.  My guess is that the at
>>>         least one of the test targets are taking steps to reduce
>>>         impact of being hard coded into so many CI tests.
>>>
>>>         *Ray Ferguson*
>>>         *Enterprise Solutions Developer*
>>>         *Devendor Tech LLC*
>>>         *ray...@de...*
>>>         <mailto:ray...@de...>
>>>         *W: 608-960-8090C: 608-516-9391*
>>>
>