#689 win32file.ReadFile seems to always return '0' as result and never winerror.ERROR_MORE_DATA

[Joos Kiener]

The documentation says:

The result is a tuple of (hr, string/PyOVERLAPPEDReadBuffer), where hr may be 0,
ERROR_MORE_DATA or ERROR_IO_PENDING.

However when calling

win32file.ReadFile(handle, 10, None)

hr is 0 and the result is the first 10 characters of the file. However the file is much longer and I would hence expect a hr of ERROR_MORE_DATA. It was expected a whole file could be read as shown below

result, buf = win32file.ReadFile(handle, 4096, None)
while result == winerror.ERROR_MORE_DATA:
result, data = win32file.ReadFile(handle, 4096, None)
buf += data
print "Hi"
return result, buf

However you never reach the while block as result is always 0.

How can one ensure that the whole file is read without having to sue a huge value for buffer?

I think this is clearly a bug or an error in the documentation.

[Mark Hammond]

A bug or error in the win32 ReadFile function or the win32file wrapper? ie, I think you'll find we are just correctly reflecting win32, for better or worse.

[Joos Kiener]

I'm not sure. Why is return value always 0 which according to MS documentations means an error occurred? Yet the file is read correctly.

[Joos Kiener]

OK. I looked at the source. I'm now sure this is a bug or at least not implementing the win32 api correctly.

Issue 1:

win32 api returns 0 for error, here 0 means success (confusing but one can live with this).

Issue 2:

from win32file.i

ok = ReadFile(hFile, buf, bufSize, &numRead, pOverlapped);
Py_END_ALLOW_THREADS
DWORD err = 0;
if (!ok) {
    err = GetLastError();
    if (err!=ERROR_MORE_DATA && err != ERROR_IO_PENDING) {
        Py_XDECREF(obRet);
        return PyWin_SetAPIError("ReadFile", err);
    }
}
if (obRet==NULL)
    obRet=PyString_FromStringAndSize((char *)buf, numRead);
else if (bBufMallocd && (numRead < bufSize))
    _PyString_Resize(&obRet, numRead);
if (obRet==NULL)
    return NULL;
return Py_BuildValue("iN", err, obRet);

The code assumes ReadFile returns ERROR_MORE_DATA if the buffer was to small. However the win32 documentation does not say this at all. It only mentions ERROR_MORE_DATA when reading from an named pipe. Just before that paragraph about pipes the documentation says

When a synchronous read operation reaches the end of a file, ReadFile returns TRUE and sets *lpNumberOfBytesRead to zero."

So in above code ok will be TRUE even when reading only 10 bytes from a 1 GB file. Thats exactly what MS documentation says. It is impossible with pywin32 win32file to determine EOF because it does not expose lpNumberOfBytesRead. So the code to check if there is more data should be:

if (ok && numRead > 0) {
    err = ERROR_MORE_DATA
}

I hope that was understandable.

EDIT:

The formatter here seems utter crap. replaces = with - WTF? Even just when using preformatted text. Can't post the code properly. Lol no, thats seems to be a firefox issue only. looks fine in IE.

[Roger Upole]

For a synchronous read, the number of bytes read is the length of the returned data.
In async mode, you have to use the overlapped api functions to find it.

[Mark Hammond]

if (ok && numRead > 0) {
err = ERROR_MORE_DATA
}

isn't correct and I don't see anything to fix here.