If read/write task is not valid, let the notifier unregister the
file descriptor. This can happen if an asynchronous socket is
used in I/O and later its file descriptor is converted to AsyncFile.
This can be "fixed" by resetting 'fd' in 'add' method, but that
would be inefficient. Instead, expect that socket is unwrapped
and then converted to asynchronous file.