We have a customer who is experiencing problems archiving data in our product - which is using the 7Zip 9.20 64bit Windows DLL. During one particular procedure, it crashes during a call to CHandler::UpdateItems and is returning 0x8003001 - STG_E_INVALIDFUNCTION.
I built a custom 7Zip with hacked in extra logging to try and see where that was coming from (including all the places in 7Zip that explicitly return that error - it's none of them).
So it seems that when the cached file tries to actually perform a seek beyond the 4Gb point; it is failing. The customer says that the file systems involved are all NTFS, so should not be subject to the FAT32 4Gb limit. It's the 64bit 7Zip so again should be able to handle large file offsets.
Can you offer any suggestions as to what is going wrong?
Many thanks.
Last edit: Duncan 2015-06-04
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1) So can you try to trace that SetSize() call?
2) Does DLL from 7-Zip 15.02 code have same problem?
3) Does 7-Zip (9.20 and 15.02) have the problem, when you archive same file?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1) I have trace in all the defined ::SetSize functions in the various classes but nothing comes out - CCacheOutStream::SetSize, COutFileStream::SetSize, COffsetOutStream::SetSize, COutMemStream::SetSize, COutMultiVolStream::SetSize. Am I missing the base somewhere?
2) We are currently waiting on the customer to try 9.38 (as the last stable release). We will try and see if they will test 15 after that. (All three versions work fine for me in my tests, but we cannot reproduce the issue anyway).
3) Unfortunately, we do not have the customer data - it is confidential; so we are having to diagnose this remotely.
Last edit: Duncan 2015-05-26
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
7-Zip returns STG_E_INVALIDFUNCTION for incorrect calls:
if (seekOrigin >= 3)
return STG_E_INVALIDFUNCTION;
I don't know why you have that code.
7z.dll calls SetSize() in client app (another module).
And that SetSize() returns STG_E_INVALIDFUNCTION. Is it so?
If so, you must debug client code (not 7z.dll code).
Also you can try 32-bit version. It must work also.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
where m_baseStream is an CComPtr<IStream>. The documentation for that (https://msdn.microsoft.com/en-us/library/windows/desktop/aa380044%28v=vs.85%29.aspx) says that a return of STG_E_INVALIDFUNCTION means
"The value of the libNewSize parameter is not supported by the implementation. Not all streams support greater than 2³² bytes. If a stream does not support more than 2³² bytes, the high DWORD data type of libNewSize must be zero. If it is nonzero, the implementation may return STG_E_INVALIDFUNCTION. In general, COM-based implementations of the IStream interface do not support streams larger than 2³² bytes."
So do I take it from that; that it is not possible to create a file greater than 4Gb using that stream class?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If some IStream implementation doesn't support 4GB+,
you can investigate why it doesn't support. Look to source code of that IStream code, if you have access to that code.
Maybe you can eliminate IStream code and use WinAPI functions SetEndOfFile() instead?
Last edit: Igor Pavlov 2015-05-26
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
IStream is just interface.
But some implementation of that interface in some class returns STG_E_INVALIDFUNCTION.
What exact class?
Is the source code of that class?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We were creating the stream with SHCreateStreamOnFileEx() - using whatever implementation that provides.
By implementing our own implementation of IStream interface using standard Win32 CreateFileEx() - and SetEndOfFile() as you suggested; it's now all working! We can create a 6Gb archive that was previously failing.
Hoorah!
Many thanks for the help and pointing me in the right direction.
Last edit: Duncan 2015-06-04
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We have a customer who is experiencing problems archiving data in our product - which is using the 7Zip 9.20 64bit Windows DLL. During one particular procedure, it crashes during a call to CHandler::UpdateItems and is returning 0x8003001 - STG_E_INVALIDFUNCTION.
I built a custom 7Zip with hacked in extra logging to try and see where that was coming from (including all the places in 7Zip that explicitly return that error - it's none of them).
This is an excerpt from the end of the logging:
Looking at my log; the issue looks to me to be the "RINOK(_stream->SetSize(newSize));" line in this function
So it seems that when the cached file tries to actually perform a seek beyond the 4Gb point; it is failing. The customer says that the file systems involved are all NTFS, so should not be subject to the FAT32 4Gb limit. It's the 64bit 7Zip so again should be able to handle large file offsets.
Can you offer any suggestions as to what is going wrong?
Many thanks.
Last edit: Duncan 2015-06-04
1) So can you try to trace that SetSize() call?
2) Does DLL from 7-Zip 15.02 code have same problem?
3) Does 7-Zip (9.20 and 15.02) have the problem, when you archive same file?
1) I have trace in all the defined ::SetSize functions in the various classes but nothing comes out - CCacheOutStream::SetSize, COutFileStream::SetSize, COffsetOutStream::SetSize, COutMemStream::SetSize, COutMultiVolStream::SetSize. Am I missing the base somewhere?
2) We are currently waiting on the customer to try 9.38 (as the last stable release). We will try and see if they will test 15 after that. (All three versions work fine for me in my tests, but we cannot reproduce the issue anyway).
3) Unfortunately, we do not have the customer data - it is confidential; so we are having to diagnose this remotely.
Last edit: Duncan 2015-05-26
7-Zip returns STG_E_INVALIDFUNCTION for incorrect calls:
I don't know why you have that code.
7z.dll calls SetSize() in client app (another module).
And that SetSize() returns STG_E_INVALIDFUNCTION. Is it so?
If so, you must debug client code (not 7z.dll code).
Also you can try 32-bit version. It must work also.
Ah! I didn't realise that it called back into our wrapper code (this is an area of the code I do not know). So that's calling:
STDMETHODIMP OurStreamWrapperClass::SetSize( UInt64 newSize ) { ULARGE_INTEGER size; size.QuadPart = newSize; return m_baseStream->SetSize( size ); }where m_baseStream is an CComPtr<IStream>. The documentation for that (https://msdn.microsoft.com/en-us/library/windows/desktop/aa380044%28v=vs.85%29.aspx) says that a return of STG_E_INVALIDFUNCTION means
"The value of the libNewSize parameter is not supported by the implementation. Not all streams support greater than 2³² bytes. If a stream does not support more than 2³² bytes, the high DWORD data type of libNewSize must be zero. If it is nonzero, the implementation may return STG_E_INVALIDFUNCTION. In general, COM-based implementations of the IStream interface do not support streams larger than 2³² bytes."
So do I take it from that; that it is not possible to create a file greater than 4Gb using that stream class?
If some IStream implementation doesn't support 4GB+,
you can investigate why it doesn't support. Look to source code of that IStream code, if you have access to that code.
Maybe you can eliminate IStream code and use WinAPI functions SetEndOfFile() instead?
Last edit: Igor Pavlov 2015-05-26
It's the standard Visual Studio platform SDK IStream - however this is built in Visual Studio 2005...
IStream is just interface.
But some implementation of that interface in some class returns STG_E_INVALIDFUNCTION.
What exact class?
Is the source code of that class?
We were creating the stream with SHCreateStreamOnFileEx() - using whatever implementation that provides.
By implementing our own implementation of IStream interface using standard Win32 CreateFileEx() - and SetEndOfFile() as you suggested; it's now all working! We can create a 6Gb archive that was previously failing.
Hoorah!
Many thanks for the help and pointing me in the right direction.
Last edit: Duncan 2015-06-04