From: <man...@us...> - 2014-09-12 10:22:28
|
Revision: 4279 http://sourceforge.net/p/modplug/code/4279 Author: manxorist Date: 2014-09-12 10:22:15 +0000 (Fri, 12 Sep 2014) Log Message: ----------- Merged revision(s) 4097-4106, 4124, 4153-4172, 4174, 4176, 4266-4267 from branches/manx/io-and-endian-fixes: [Ref] Add mpt::true_type and mpt::false_type. This simplifies defining our own type traits. For compilers which support for <type_traits>, these are the same as std::true_type and std::false_type, for other compilers a simple ad-hoc C++11 compatible version is used. ........ [Ref] Add mpt::is_binary_safe<T> which shall be true for types which have a defined-endian and defined-packing in-memory representationand and can thus be directly written into binary files. ........ [Ref] Add mptIO.h. This contains a small wrapper that allows for a common interface for both, FILE* and std::ostream (and potentially anything else) file i/o, and provides small, endian-safe, writing helper functions. ........ [Fix] Fix a lot of endianness problems in WriteInstrumentHeaderStructOrField() and SaveIT(). The full test suite still does not pass on big-endian, so this is just another tiny step. ........ [Fix] In SaveModularInstrumentData, modularInstSize gets endian-swapped on big-endian and used after that without swapping it back. Fix it by using endian-safe writing function. ........ [Imp] Run the full test suite (including file saving) for libopenmpt on big-endian platforms. It finally passes. ........ [Ref] Add mpt::IO::WriteConvertEndianness(). ........ [Fix] Fix mpt::IO::WriteConvertEndianness(). ........ [Fix] Simplify mpt::IO and make it actually work with std::ostream. ........ [Fix] Add mptIO.h to MSVC projects. ........ [Fix] Fix mpt::IO::Flush(FILE*). ........ [Ref] Add read support to mpt::IO. [Ref] Use 64bit FILE* on MSVC and posix systems. Add static asserts to force 64bit file IO (breaks obscure platforms for now). ........ [Ref] mptIO: Silence stupid MSVC warnings. [Ref] mptIO: Add ReadBinaryLE and WriteBinaryLE. [Ref] mptIO: Add ReadBinaryTruncatedLE and WriteBinaryTruncatedLE. ........ [Ref] serialization_utils: Base Binaryread and Binaryrite on mpt::IO. ........ [Ref] Add byte swapping for 64bit integers. ........ [Ref] mptIO: Add adaptive-size integer (as used by serialization_utils) read and write functions. ........ [Ref] serialization_utils: Convert to mpt::IO adaptive integer I/O functions. ........ [Ref] serialization_utils: Remove unused ReadItem specializations for reading floats and doubles of the opposite size. These are not used anywhere in the code. The one single place where floating point data is read and written via serialization_utils, there is no explicit size tag at all and 32bit is assumed. ........ [Fix] mptIO: Compile fix. ........ [Ref] serialization_utils: Binaryread and Binaryrite are only ever used to write plain integers or plain floats. Enforce this via compile-time assertions. [Fix] serialization_utils: Use explicit IEEE floating point format. ........ [Ref] mptIO: Kill unused ReadBinaryLE and WriteBinaryLE. ........ [Ref] serialization_utils: Replace all usage sites of srlztn::Binarywrite and srlztn::Binaryread outside of serialization_utils with mpt::IO. ........ [Ref] serialization_utils: Document insanity in historic serialization encoding changes. ........ [Ref] mptIO: Add minSize parameter for WriteAdaptiveInt*. ........ [Fix] serialization_utils: Fix writing of map entry count values greater than 16383. ........ [Ref] mptIO: Kill WriteBinaryTruncatedLE. ........ [Fix] __builtin_bswap16 is only available since GCC 4.8 . ........ [Ref] mptIO: Remove bogus code. ........ [Fix] serialization_utils: Fix comment about change in written on-disk format. ........ [Ref] mptIO: Add maxSize parameter for WriteAdaptiveInt*. ........ [Fix] serialization_utils: Revert r4166 and write fixed size=2 AdaptiveInt64LE again for map entry count. (thanks to relabs for noticing) [Fix] serialization_utils: Actually explain the 16000 entries limit and fix the comments about the fixed-size AdaptiveInt64LE usages. (thanks to relabs for explaining) ........ [Ref] serialization_utils: Remove unused functions from public interface. [Fix] serialization_utils: Make debug output endian-clean. ........ [Ref] serialization_utils: Clarify comment once again. ........ [Fix] Do not redefine _FILE_OFFSET_BITS if already defined. ........ [Fix] mptIO: Add back support for 32 bit file offsets in FILE* and std::fstream. ........ Revision Links: -------------- http://sourceforge.net/p/modplug/code/4166 Modified Paths: -------------- trunk/OpenMPT/common/BuildSettings.h trunk/OpenMPT/common/Endianness.h trunk/OpenMPT/common/serialization_utils.cpp trunk/OpenMPT/common/serialization_utils.h trunk/OpenMPT/common/stdafx.h trunk/OpenMPT/common/typedefs.h trunk/OpenMPT/libopenmpt/libopenmpt.vcxproj trunk/OpenMPT/libopenmpt/libopenmpt.vcxproj.filters trunk/OpenMPT/libopenmpt/libopenmptDLL.vcxproj trunk/OpenMPT/libopenmpt/libopenmptDLL.vcxproj.filters trunk/OpenMPT/libopenmpt/libopenmpt_test.vcxproj trunk/OpenMPT/libopenmpt/libopenmpt_test.vcxproj.filters trunk/OpenMPT/mptrack/mptrack_08.vcproj trunk/OpenMPT/mptrack/mptrack_10.vcxproj trunk/OpenMPT/mptrack/mptrack_10.vcxproj.filters trunk/OpenMPT/soundlib/Load_it.cpp trunk/OpenMPT/soundlib/ModSequence.cpp trunk/OpenMPT/soundlib/Sndfile.cpp trunk/OpenMPT/soundlib/pattern.cpp trunk/OpenMPT/soundlib/tuning.cpp Added Paths: ----------- trunk/OpenMPT/common/mptIO.h Property Changed: ---------------- trunk/OpenMPT/ Index: trunk/OpenMPT =================================================================== --- trunk/OpenMPT 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT 2014-09-12 10:22:15 UTC (rev 4279) Property changes on: trunk/OpenMPT ___________________________________________________________________ Modified: svn:mergeinfo ## -3,6 +3,7 ## /branches/manx/gcc-fixes:2018-2022,2024-2048,2052-2071,2075-2077,2080,2087-2089 /branches/manx/header-dependencies-cleanups:1394-1397,1401-1402,1405-1406 /branches/manx/instrument-fields-fixes:2203-2238 +/branches/manx/io-and-endian-fixes:4097-4267 /branches/manx/lossy-export:2653-2654,2660,2664,2678,2681 /branches/manx/lossy-export-xiph:2663,2677,2680 /branches/manx/mptstring-stdstring-support:2204,2208,2212,2214,2217,2220,2224,2259,2261-2262,2264,2267 \ No newline at end of property Modified: trunk/OpenMPT/common/BuildSettings.h =================================================================== --- trunk/OpenMPT/common/BuildSettings.h 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/common/BuildSettings.h 2014-09-12 10:22:15 UTC (rev 4279) @@ -250,10 +250,6 @@ #define NO_MINIZ #endif -#if defined(MPT_PLATFORM_BIG_ENDIAN) && !defined(MODPLUG_NO_FILESAVE) -#define MODPLUG_NO_FILESAVE // file saving is broken on big endian -#endif - #if !defined(MPT_CHARSET_WIN32) && !defined(MPT_CHARSET_ICONV) && !defined(MPT_CHARSET_CODECVTUTF8) && !defined(MPT_CHARSET_INTERNAL) #define MPT_CHARSET_INTERNAL #endif @@ -378,6 +374,10 @@ #endif +#ifndef _FILE_OFFSET_BITS +#define _FILE_OFFSET_BITS 64 +#endif + #if MPT_COMPILER_MSVC #ifndef _CRT_SECURE_NO_WARNINGS #define _CRT_SECURE_NO_WARNINGS // Define to disable the "This function or variable may be unsafe" warnings. Modified: trunk/OpenMPT/common/Endianness.h =================================================================== --- trunk/OpenMPT/common/Endianness.h 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/common/Endianness.h 2014-09-12 10:22:15 UTC (rev 4279) @@ -27,12 +27,17 @@ // format. #if MPT_COMPILER_GCC +#if MPT_GCC_AT_LEAST(4,8,0) +#define bswap16 __builtin_bswap16 +#endif #if MPT_GCC_AT_LEAST(4,3,0) #define MPT_bswap32 __builtin_bswap32 +#define MPT_bswap64 __builtin_bswap64 #endif #elif MPT_COMPILER_MSVC #define MPT_bswap16 _byteswap_ushort #define MPT_bswap32 _byteswap_ulong +#define MPT_bswap64 _byteswap_uint64 #endif // catch system macros @@ -46,6 +51,11 @@ #define MPT_bswap32 bswap32 #endif #endif +#ifndef MPT_bswap64 +#ifdef bswap64 +#define MPT_bswap64 bswap64 +#endif +#endif // No intrinsics available #ifndef MPT_bswap16 @@ -54,7 +64,22 @@ #ifndef MPT_bswap32 #define MPT_bswap32(x) (((x & 0xFF) << 24) | ((x & 0xFF00) << 8) | ((x & 0xFF0000) >> 8) | ((x & 0xFF000000) >> 24)) #endif +#ifndef MPT_bswap64 +#define MPT_bswap64(x) \ + ( uint64(0) \ + | (((x >> 0) & 0xff) << 56) \ + | (((x >> 8) & 0xff) << 48) \ + | (((x >> 16) & 0xff) << 40) \ + | (((x >> 24) & 0xff) << 32) \ + | (((x >> 32) & 0xff) << 24) \ + | (((x >> 40) & 0xff) << 16) \ + | (((x >> 48) & 0xff) << 8) \ + | (((x >> 56) & 0xff) << 0) \ + ) \ +/**/ +#endif + // Deprecated. Use "SwapBytesXX" versions below. #ifdef MPT_PLATFORM_BIG_ENDIAN inline uint32 LittleEndian(uint32 x) { return MPT_bswap32(x); } @@ -69,23 +94,31 @@ #endif #if defined(MPT_PLATFORM_BIG_ENDIAN) +#define MPT_bswap64le(x) MPT_bswap64(x) #define MPT_bswap32le(x) MPT_bswap32(x) #define MPT_bswap16le(x) MPT_bswap16(x) +#define MPT_bswap64be(x) (x) #define MPT_bswap32be(x) (x) #define MPT_bswap16be(x) (x) #elif defined(MPT_PLATFORM_LITTLE_ENDIAN) +#define MPT_bswap64be(x) MPT_bswap64(x) #define MPT_bswap32be(x) MPT_bswap32(x) #define MPT_bswap16be(x) MPT_bswap16(x) +#define MPT_bswap64le(x) (x) #define MPT_bswap32le(x) (x) #define MPT_bswap16le(x) (x) #endif +inline uint64 SwapBytesBE_(uint64 value) { return MPT_bswap64be(value); } inline uint32 SwapBytesBE_(uint32 value) { return MPT_bswap32be(value); } inline uint16 SwapBytesBE_(uint16 value) { return MPT_bswap16be(value); } +inline uint64 SwapBytesLE_(uint64 value) { return MPT_bswap64le(value); } inline uint32 SwapBytesLE_(uint32 value) { return MPT_bswap32le(value); } inline uint16 SwapBytesLE_(uint16 value) { return MPT_bswap16le(value); } +inline int64 SwapBytesBE_(int64 value) { return MPT_bswap64be(value); } inline int32 SwapBytesBE_(int32 value) { return MPT_bswap32be(value); } inline int16 SwapBytesBE_(int16 value) { return MPT_bswap16be(value); } +inline int64 SwapBytesLE_(int64 value) { return MPT_bswap64le(value); } inline int32 SwapBytesLE_(int32 value) { return MPT_bswap32le(value); } inline int16 SwapBytesLE_(int16 value) { return MPT_bswap16le(value); } @@ -116,10 +149,13 @@ #undef MPT_bswap16le #undef MPT_bswap32le +#undef MPT_bswap64le #undef MPT_bswap16be #undef MPT_bswap32be +#undef MPT_bswap64be #undef MPT_bswap16 #undef MPT_bswap32 +#undef MPT_bswap64 // 1.0f --> 0x3f800000u @@ -230,6 +266,13 @@ } }; +namespace mpt { + +template <> struct is_binary_safe<IEEE754binary32Emulated<0,1,2,3> > : public mpt::true_type { }; +template <> struct is_binary_safe<IEEE754binary32Emulated<3,2,1,0> > : public mpt::true_type { }; + +} // namespace mpt + #if MPT_PLATFORM_IEEE_FLOAT struct IEEE754binary32Native @@ -298,6 +341,12 @@ } }; +namespace mpt { + +template <> struct is_binary_safe<IEEE754binary32Native> : public mpt::true_type { }; + +} // namespace mpt + #if defined(MPT_PLATFORM_LITTLE_ENDIAN) typedef IEEE754binary32Native IEEE754binary32LE; typedef IEEE754binary32Emulated<0,1,2,3> IEEE754binary32BE; Copied: trunk/OpenMPT/common/mptIO.h (from rev 4267, branches/manx/io-and-endian-fixes/common/mptIO.h) =================================================================== --- trunk/OpenMPT/common/mptIO.h (rev 0) +++ trunk/OpenMPT/common/mptIO.h 2014-09-12 10:22:15 UTC (rev 4279) @@ -0,0 +1,348 @@ +/* + * mptIO.h + * ------- + * Purpose: Basic functions for reading/writing binary and endian safe data to/from files/streams. + * Notes : This is work-in-progress. + * Some useful functions for reading and writing are still missing. + * Authors: Joern Heusipp + * The OpenMPT source code is released under the BSD license. Read LICENSE for more details. + */ + + +#pragma once + + +#include "../common/typedefs.h" +#include "../common/Endianness.h" +#include <ios> +#include <iostream> +#include <istream> +#include <limits> +#include <ostream> +#if defined(HAS_TYPE_TRAITS) +#include <type_traits> +#endif +#include <cstdio> +#include <cstring> +#include <stdio.h> + + +OPENMPT_NAMESPACE_BEGIN + + +namespace mpt { + +namespace IO { + +typedef int64 Offset; + + + +// Returns true iff 'off' fits into 'Toff'. +template < typename Toff > +inline bool OffsetFits(IO::Offset off) +{ + return (static_cast<IO::Offset>(mpt::saturate_cast<Toff>(off)) == off); +} + + + +//STATIC_ASSERT(sizeof(std::streamoff) == 8); // Assert 64bit file support. +inline bool IsValid(std::ostream & f) { return !f.fail(); } +inline bool IsValid(std::istream & f) { return !f.fail(); } +inline bool IsValid(std::iostream & f) { return !f.fail(); } +inline IO::Offset TellRead(std::istream & f) { return f.tellg(); } +inline IO::Offset TellWrite(std::ostream & f) { return f.tellp(); } +inline bool SeekBegin(std::ostream & f) { f.seekp(0); return !f.fail(); } +inline bool SeekBegin(std::istream & f) { f.seekg(0); return !f.fail(); } +inline bool SeekBegin(std::iostream & f) { f.seekg(0); f.seekp(0); return !f.fail(); } +inline bool SeekEnd(std::ostream & f) { f.seekp(0, std::ios::end); return !f.fail(); } +inline bool SeekEnd(std::istream & f) { f.seekg(0, std::ios::end); return !f.fail(); } +inline bool SeekEnd(std::iostream & f) { f.seekg(0, std::ios::end); f.seekp(0, std::ios::end); return !f.fail(); } +inline bool SeekAbsolute(std::ostream & f, IO::Offset pos) { if(!OffsetFits<std::streamoff>(pos)) { return false; } f.seekp(pos, std::ios::beg); return !f.fail(); } +inline bool SeekAbsolute(std::istream & f, IO::Offset pos) { if(!OffsetFits<std::streamoff>(pos)) { return false; } f.seekg(pos, std::ios::beg); return !f.fail(); } +inline bool SeekAbsolute(std::iostream & f, IO::Offset pos) { if(!OffsetFits<std::streamoff>(pos)) { return false; } f.seekg(pos, std::ios::beg); f.seekp(pos, std::ios::beg); return !f.fail(); } +inline bool SeekRelative(std::ostream & f, IO::Offset off) { if(!OffsetFits<std::streamoff>(off)) { return false; } f.seekp(off, std::ios::cur); return !f.fail(); } +inline bool SeekRelative(std::istream & f, IO::Offset off) { if(!OffsetFits<std::streamoff>(off)) { return false; } f.seekg(off, std::ios::cur); return !f.fail(); } +inline bool SeekRelative(std::iostream & f, IO::Offset off) { if(!OffsetFits<std::streamoff>(off)) { return false; } f.seekg(off, std::ios::cur); f.seekp(off, std::ios::cur); return !f.fail(); } +inline IO::Offset ReadRaw(std::istream & f, uint8 * data, std::size_t size) { return f.read(reinterpret_cast<char *>(data), size) ? f.gcount() : std::streamsize(0); } +inline IO::Offset ReadRaw(std::istream & f, char * data, std::size_t size) { return f.read(data, size) ? f.gcount() : std::streamsize(0); } +inline IO::Offset ReadRaw(std::istream & f, void * data, std::size_t size) { return f.read(reinterpret_cast<char *>(data), size) ? f.gcount() : std::streamsize(0); } +inline bool WriteRaw(std::ostream & f, const uint8 * data, std::size_t size) { f.write(reinterpret_cast<const char *>(data), size); return !f.fail(); } +inline bool WriteRaw(std::ostream & f, const char * data, std::size_t size) { f.write(data, size); return !f.fail(); } +inline bool WriteRaw(std::ostream & f, const void * data, std::size_t size) { f.write(reinterpret_cast<const char *>(data), size); return !f.fail(); } +inline bool IsEof(std::istream & f) { return f.eof(); } +inline bool Flush(std::ostream & f) { f.flush(); return !f.fail(); } + + + +inline bool IsValid(FILE* & f) { return f != NULL; } + +#if MPT_COMPILER_MSVC + +inline IO::Offset TellRead(FILE* & f) { return _ftelli64(f); } +inline IO::Offset TellWrite(FILE* & f) { return _ftelli64(f); } +inline bool SeekBegin(FILE* & f) { return _fseeki64(f, 0, SEEK_SET) == 0; } +inline bool SeekEnd(FILE* & f) { return _fseeki64(f, 0, SEEK_END) == 0; } +inline bool SeekAbsolute(FILE* & f, IO::Offset pos) { return _fseeki64(f, pos, SEEK_SET) == 0; } +inline bool SeekRelative(FILE* & f, IO::Offset off) { return _fseeki64(f, off, SEEK_CUR) == 0; } + +#elif defined(_POSIX_SOURCE) && (_POSIX_SOURCE > 0) + +//STATIC_ASSERT(sizeof(off_t) == 8); +inline IO::Offset TellRead(FILE* & f) { return ftello(f); } +inline IO::Offset TellWrite(FILE* & f) { return ftello(f); } +inline bool SeekBegin(FILE* & f) { return fseeko(f, 0, SEEK_SET) == 0; } +inline bool SeekEnd(FILE* & f) { return fseeko(f, 0, SEEK_END) == 0; } +inline bool SeekAbsolute(FILE* & f, IO::Offset pos) { return OffsetFits<off_t>(pos) && (fseek(f, mpt::saturate_cast<off_t>(pos), SEEK_SET) == 0); } +inline bool SeekRelative(FILE* & f, IO::Offset off) { return OffsetFits<off_t>(off) && (fseek(f, mpt::saturate_cast<off_t>(off), SEEK_CUR) == 0); } + +#else + +//STATIC_ASSERT(sizeof(long) == 8); // Fails on 32bit non-POSIX systems for now. +inline IO::Offset TellRead(FILE* & f) { return ftell(f); } +inline IO::Offset TellWrite(FILE* & f) { return ftell(f); } +inline bool SeekBegin(FILE* & f) { return fseek(f, 0, SEEK_SET) == 0; } +inline bool SeekEnd(FILE* & f) { return fseek(f, 0, SEEK_END) == 0; } +inline bool SeekAbsolute(FILE* & f, IO::Offset pos) { return OffsetFits<long>(pos) && (fseek(f, mpt::saturate_cast<long>(pos), SEEK_SET) == 0); } +inline bool SeekRelative(FILE* & f, IO::Offset off) { return OffsetFits<long>(off) && (fseek(f, mpt::saturate_cast<long>(off), SEEK_CUR) == 0); } + +#endif + +inline IO::Offset ReadRaw(FILE * & f, uint8 * data, std::size_t size) { return fread(data, 1, size, f); } +inline IO::Offset ReadRaw(FILE * & f, char * data, std::size_t size) { return fread(data, 1, size, f); } +inline IO::Offset ReadRaw(FILE * & f, void * data, std::size_t size) { return fread(data, 1, size, f); } +inline bool WriteRaw(FILE* & f, const uint8 * data, std::size_t size) { return fwrite(data, 1, size, f) == size; } +inline bool WriteRaw(FILE* & f, const char * data, std::size_t size) { return fwrite(data, 1, size, f) == size; } +inline bool WriteRaw(FILE* & f, const void * data, std::size_t size) { return fwrite(data, 1, size, f) == size; } +inline bool IsEof(FILE * & f) { return feof(f) != 0; } +inline bool Flush(FILE* & f) { return fflush(f) == 0; } + + + +template <typename Tbinary, typename Tfile> +inline bool Read(Tfile & f, Tbinary & v) +{ + return IO::ReadRaw(f, mpt::GetRawBytes(v), sizeof(Tbinary)) == sizeof(Tbinary); +} + +template <typename Tbinary, typename Tfile> +inline bool Write(Tfile & f, const Tbinary & v) +{ + return IO::WriteRaw(f, mpt::GetRawBytes(v), sizeof(Tbinary)); +} + +template <typename T, typename Tfile> +inline bool ReadBinaryTruncatedLE(Tfile & f, T & v, std::size_t size) +{ + bool result = false; + #ifdef HAS_TYPE_TRAITS + static_assert(std::is_trivial<T>::value == true, ""); + #endif + uint8 bytes[sizeof(T)]; + std::memset(bytes, 0, sizeof(T)); + result = IO::ReadRaw(f, bytes, std::min(size, sizeof(T))) == std::min(size, sizeof(T)); + #ifdef MPT_PLATFORM_BIG_ENDIAN + std::reverse(bytes, bytes + sizeof(T)); + #endif + std::memcpy(&v, bytes, sizeof(T)); + return result; +} + +template <typename T, typename Tfile> +inline bool ReadIntLE(Tfile & f, T & v) +{ + bool result = false; + STATIC_ASSERT(std::numeric_limits<T>::is_integer); + uint8 bytes[sizeof(T)]; + std::memset(bytes, 0, sizeof(T)); + result = (IO::ReadRaw(f, bytes, sizeof(T)) == sizeof(T)); + T val = 0; + std::memcpy(&val, bytes, sizeof(T)); + v = SwapBytesReturnLE(val); + return result; +} + +template <typename T, typename Tfile> +inline bool ReadIntBE(Tfile & f, T & v) +{ + bool result = false; + STATIC_ASSERT(std::numeric_limits<T>::is_integer); + uint8 bytes[sizeof(T)]; + std::memset(bytes, 0, sizeof(T)); + result = (IO::ReadRaw(f, bytes, sizeof(T)) == sizeof(T)); + T val = 0; + std::memcpy(&val, bytes, sizeof(T)); + v = SwapBytesReturnBE(val); + return result; +} + +template <typename Tfile> +inline bool ReadAdaptiveInt16LE(Tfile & f, uint16 & v) +{ + bool result = true; + uint8 byte = 0; + std::size_t additionalBytes = 0; + v = 0; + byte = 0; + if(!IO::ReadIntLE<uint8>(f, byte)) result = false; + additionalBytes = (byte & 0x01); + v = byte >> 1; + for(std::size_t i = 0; i < additionalBytes; ++i) + { + byte = 0; + if(!IO::ReadIntLE<uint8>(f, byte)) result = false; + v |= (static_cast<uint16>(byte) << (((i+1)*8) - 1)); + } + return result; +} + +template <typename Tfile> +inline bool ReadAdaptiveInt32LE(Tfile & f, uint32 & v) +{ + bool result = true; + uint8 byte = 0; + std::size_t additionalBytes = 0; + v = 0; + byte = 0; + if(!IO::ReadIntLE<uint8>(f, byte)) result = false; + additionalBytes = (byte & 0x03); + v = byte >> 2; + for(std::size_t i = 0; i < additionalBytes; ++i) + { + byte = 0; + if(!IO::ReadIntLE<uint8>(f, byte)) result = false; + v |= (static_cast<uint32>(byte) << (((i+1)*8) - 2)); + } + return result; +} + +template <typename Tfile> +inline bool ReadAdaptiveInt64LE(Tfile & f, uint64 & v) +{ + bool result = true; + uint8 byte = 0; + std::size_t additionalBytes = 0; + v = 0; + byte = 0; + if(!IO::ReadIntLE<uint8>(f, byte)) result = false; + additionalBytes = (1 << (byte & 0x03)) - 1; + v = byte >> 2; + for(std::size_t i = 0; i < additionalBytes; ++i) + { + byte = 0; + if(!IO::ReadIntLE<uint8>(f, byte)) result = false; + v |= (static_cast<uint64>(byte) << (((i+1)*8) - 2)); + } + return result; +} + +template <typename T, typename Tfile> +inline bool WriteIntLE(Tfile & f, const T & v) +{ + STATIC_ASSERT(std::numeric_limits<T>::is_integer); + const T val = SwapBytesReturnLE(v); + uint8 bytes[sizeof(T)]; + std::memcpy(bytes, &val, sizeof(T)); + return IO::WriteRaw(f, bytes, sizeof(T)); +} + +template <typename T, typename Tfile> +inline bool WriteIntBE(Tfile & f, const T & v) +{ + STATIC_ASSERT(std::numeric_limits<T>::is_integer); + const T val = SwapBytesReturnBE(v); + uint8 bytes[sizeof(T)]; + std::memcpy(bytes, &val, sizeof(T)); + return IO::WriteRaw(f, bytes, sizeof(T)); +} + +template <typename Tfile> +inline bool WriteAdaptiveInt16LE(Tfile & f, const uint16 & v, std::size_t minSize = 0, std::size_t maxSize = 0) +{ + ASSERT(minSize == 0 || minSize == 1 || minSize == 2); + ASSERT(maxSize == 0 || maxSize == 1 || maxSize == 2); + ASSERT(maxSize == 0 || maxSize >= minSize); + if(v < 0x80 && minSize <= 1 && (1 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint8>(f, static_cast<uint8>(v << 1) | 0x00); + } else if(v < 0x8000 && minSize <= 2 && (2 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint16>(f, static_cast<uint16>(v << 1) | 0x01); + } else + { + ASSERT(false); + return false; + } +} + +template <typename Tfile> +inline bool WriteAdaptiveInt32LE(Tfile & f, const uint32 & v, std::size_t minSize = 0, std::size_t maxSize = 0) +{ + ASSERT(minSize == 0 || minSize == 1 || minSize == 2 || minSize == 3 || minSize == 4); + ASSERT(maxSize == 0 || maxSize == 1 || maxSize == 2 || maxSize == 3 || maxSize == 4); + ASSERT(maxSize == 0 || maxSize >= minSize); + if(v < 0x40 && minSize <= 1 && (1 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint8>(f, static_cast<uint8>(v << 2) | 0x00); + } else if(v < 0x4000 && minSize <= 2 && (2 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint16>(f, static_cast<uint16>(v << 2) | 0x01); + } else if(v < 0x400000 && minSize <= 3 && (3 <= maxSize || maxSize == 0)) + { + uint32 value = static_cast<uint32>(v << 2) | 0x02; + uint8 bytes[3]; + bytes[0] = static_cast<uint8>(value >> 0); + bytes[1] = static_cast<uint8>(value >> 8); + bytes[2] = static_cast<uint8>(value >> 16); + return IO::WriteRaw(f, bytes, 3); + } else if(v < 0x40000000 && minSize <= 4 && (4 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint32>(f, static_cast<uint32>(v << 2) | 0x03); + } else + { + ASSERT(false); + return false; + } +} + +template <typename Tfile> +inline bool WriteAdaptiveInt64LE(Tfile & f, const uint64 & v, std::size_t minSize = 0, std::size_t maxSize = 0) +{ + ASSERT(minSize == 0 || minSize == 1 || minSize == 2 || minSize == 4 || minSize == 8); + ASSERT(maxSize == 0 || maxSize == 1 || maxSize == 2 || maxSize == 4 || maxSize == 8); + ASSERT(maxSize == 0 || maxSize >= minSize); + if(v < 0x40 && minSize <= 1 && (1 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint8>(f, static_cast<uint8>(v << 2) | 0x00); + } else if(v < 0x4000 && minSize <= 2 && (2 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint16>(f, static_cast<uint16>(v << 2) | 0x01); + } else if(v < 0x40000000 && minSize <= 4 && (4 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint32>(f, static_cast<uint32>(v << 2) | 0x02); + } else if(v < 0x4000000000000000ull && minSize <= 8 && (8 <= maxSize || maxSize == 0)) + { + return IO::WriteIntLE<uint64>(f, static_cast<uint64>(v << 2) | 0x03); + } else + { + ASSERT(false); + return false; + } +} + +template <typename T, typename Tfile> +inline bool WriteConvertEndianness(Tfile & f, T & v) +{ + v.ConvertEndianness(); + bool result = IO::WriteRaw(f, reinterpret_cast<const uint8 *>(&v), sizeof(T)); + v.ConvertEndianness(); + return result; +} + +} // namespace IO + +} // namespace mpt + + +OPENMPT_NAMESPACE_END Modified: trunk/OpenMPT/common/serialization_utils.cpp =================================================================== --- trunk/OpenMPT/common/serialization_utils.cpp 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/common/serialization_utils.cpp 2014-09-12 10:22:15 UTC (rev 4279) @@ -50,131 +50,17 @@ return true; } - -static uint8 GetByteReq1248(const uint64 size) -//-------------------------------------------- -{ - if((size >> 6) == 0) return 1; - if((size >> (1*8+6)) == 0) return 2; - if((size >> (3*8+6)) == 0) return 4; - return 8; -} - - -static uint8 GetByteReq1234(const uint32 num) -//------------------------------------------- -{ - if((num >> 6) == 0) return 1; - if((num >> (1*8+6)) == 0) return 2; - if((num >> (2*8+6)) == 0) return 3; - return 4; -} - - -void WriteAdaptive12(std::ostream& oStrm, const uint16 num) -//--------------------------------------------------------- -{ - if(num >> 7 == 0) - Binarywrite<uint16>(oStrm, num << 1, 1); - else - Binarywrite<uint16>(oStrm, (num << 1) | 1); -} - - -void WriteAdaptive1234(std::ostream& oStrm, const uint32 num) -//----------------------------------------------------------- -{ - const uint8 bc = GetByteReq1234(num); - const uint32 sizeInstruction = (num << 2) | (bc - 1); - Binarywrite<uint32>(oStrm, sizeInstruction, bc); -} - - //Format: First bit tells whether the size indicator is 1 or 2 bytes. static void WriteAdaptive12String(std::ostream& oStrm, const std::string& str) //---------------------------------------------------------------------------- { uint16 s = static_cast<uint16>(str.size()); LimitMax(s, uint16(uint16_max / 2)); - WriteAdaptive12(oStrm, s); + mpt::IO::WriteAdaptiveInt16LE(oStrm, s); oStrm.write(str.c_str(), s); } -// Works only for arguments 1,2,4,8 -static uint8 Log2(const uint8& val) -//--------------------------------- -{ - if(val == 1) return 0; - else if(val == 2) return 1; - else if(val == 4) return 2; - else return 3; -} - - -void WriteAdaptive1248(std::ostream& oStrm, const uint64& num) -//------------------------------------------------------------ -{ - const uint8 bc = GetByteReq1248(num); - const uint64 sizeInstruction = (num << 2) | Log2(bc); - Binarywrite<uint64>(oStrm, sizeInstruction, bc); -} - - -void ReadAdaptive12(std::istream& iStrm, uint16& val) -//--------------------------------------------------- -{ - Binaryread<uint16>(iStrm, val, 1); - if(val & 1) - { - uint8 hi = 0; - Binaryread(iStrm, hi); - val &= 0xff; - val |= hi << 8; - } - val >>= 1; -} - - -void ReadAdaptive1234(std::istream& iStrm, uint32& val) -//----------------------------------------------------- -{ - Binaryread<uint32>(iStrm, val, 1); - const uint8 bc = 1 + static_cast<uint8>(val & 3); - uint8 v2 = 0; - uint8 v3 = 0; - uint8 v4 = 0; - if(bc >= 2) Binaryread(iStrm, v2); - if(bc >= 3) Binaryread(iStrm, v3); - if(bc >= 4) Binaryread(iStrm, v4); - val &= 0xff; - val |= (v2 << 8) | (v3 << 16) | (v4 << 24); - val >>= 2; -} - - -void ReadAdaptive1248(std::istream& iStrm, uint64& val) -//----------------------------------------------------- -{ - Binaryread<uint64>(iStrm, val, 1); - uint8 bc = 1 << static_cast<uint8>(val & 3); - int byte = 1; - val &= 0xff; - while(bc > 1) - { - uint8 v = 0; - Binaryread(iStrm, v); - if(byte < 8) - { - val |= uint64(v) << (byte*8); - } - byte++; - bc--; - } - val >>= 2; -} - - void WriteItemString(std::ostream& oStrm, const char* const pStr, const size_t nSize) //-------------------------------------------------------------------------------- { @@ -227,10 +113,11 @@ std::string str; if (IsPrintableId(pId, nLength)) std::copy(pId, pId + nLength, std::back_inserter<std::string>(str)); - else if (nLength <= 4) // Interpret ID as integer value. + else if (nLength <= 8) // Interpret ID as integer value. { - int32 val = 0; - memcpy(&val, pId, nLength); + int64 val = 0; + std::memcpy(&val, pId, nLength); + val = SwapBytesReturnLE(val); str = Stringify(val); } return str; @@ -379,16 +266,16 @@ { AddWriteNote(SNW_CHANGING_IDSIZE_WITH_FIXED_IDSIZESETTING); return; } if (m_nIdbytes == IdSizeVariable) //Variablesize ID? - WriteAdaptive12(mapStream, static_cast<uint16>(nIdSize)); + mpt::IO::WriteAdaptiveInt16LE(mapStream, static_cast<uint16>(nIdSize)); if(nIdSize > 0) mapStream.write(pId, nIdSize); } if (GetFlag(RwfWMapStartPosEntry)) //Startpos - WriteAdaptive1248(mapStream, rposDataStart); + mpt::IO::WriteAdaptiveInt64LE(mapStream, rposDataStart); if (GetFlag(RwfWMapSizeEntry)) //Entrysize - WriteAdaptive1248(mapStream, nDatasize); + mpt::IO::WriteAdaptiveInt64LE(mapStream, nDatasize); if (GetFlag(RwfWMapDescEntry)) //Entry descriptions WriteAdaptive12String(mapStream, std::string(pszDesc)); @@ -457,15 +344,15 @@ const uint8 flags = tempU8; if(flags != s_DefaultFlagbyte) { - WriteAdaptive1234(oStrm, 2); //Headersize - now it is 2. + mpt::IO::WriteAdaptiveInt32LE(oStrm, 2); //Headersize - now it is 2. Binarywrite<uint8>(oStrm, HeaderId_FlagByte); Binarywrite<uint8>(oStrm, flags); } else - WriteAdaptive1234(oStrm, 0); + mpt::IO::WriteAdaptiveInt32LE(oStrm, 0); if(Testbit(header, 4)) // Version(numeric)? - WriteAdaptive1248(oStrm, nVersion); + mpt::IO::WriteAdaptiveInt64LE(oStrm, nVersion); if(Testbit(flags, 0)) // Custom IDbytecount? { @@ -474,7 +361,7 @@ } if(Testbit(flags, 1)) // Fixedsize entries? - WriteAdaptive1234(oStrm, m_nFixedEntrySize); + mpt::IO::WriteAdaptiveInt32LE(oStrm, m_nFixedEntrySize); //Entrycount. Reserve two bytes(max uint16_max / 4 entries), actual value is written after writing data. m_posEntrycount = oStrm.tellp(); @@ -612,7 +499,7 @@ // Read headerdata size uint32 tempU32 = 0; - ReadAdaptive1234(iStrm, tempU32); + mpt::IO::ReadAdaptiveInt32LE(iStrm, tempU32); const uint32 headerdatasize = tempU32; // If headerdatasize != 0, read known headerdata and ignore rest. @@ -631,7 +518,7 @@ // Read version numeric if available. if (Testbit(header, 4)) { - ReadAdaptive1248(iStrm, tempU64); + mpt::IO::ReadAdaptiveInt64LE(iStrm, tempU64); m_nReadVersion = tempU64; if(tempU64 > nVersion) AddReadNote(SNR_LOADING_OBJECT_WITH_LARGER_VERSION); @@ -656,7 +543,7 @@ m_nFixedEntrySize = 0; if(Testbit(flagbyte, 1)) // Fixedsize entries? - ReadAdaptive1234(iStrm, m_nFixedEntrySize); + mpt::IO::ReadAdaptiveInt32LE(iStrm, m_nFixedEntrySize); SetFlag(RwfRMapHasStartpos, Testbit(header, 2)); SetFlag(RwfRMapHasSize, Testbit(header, 3)); @@ -670,7 +557,7 @@ if (Testbit(flagbyte, 2)) // Object description? { uint16 size = 0; - ReadAdaptive12(iStrm, size); + mpt::IO::ReadAdaptiveInt16LE(iStrm, size); iStrm.ignore(size * (GetFlag(RwfRTwoBytesDescChar) ? 2 : 1)); } @@ -678,8 +565,10 @@ iStrm.ignore(5); // Read entrycount - ReadAdaptive1248(iStrm, tempU64); - if(tempU64 > 16000) // FIXME: 16000 appear like a totally arbitrary limit. May be to avoid out-of-memory DoS. + mpt::IO::ReadAdaptiveInt64LE(iStrm, tempU64); + if(tempU64 > 16000) + // The current code can only write 16383 entries because it uses a Adaptive64LE with a fixed size=2 + // Additionally, 16000 is an arbitrary limit to avoid an out-of-memory DoS when caching the map. { AddReadNote(SNR_TOO_MANY_ENTRIES_TO_READ); return; } m_nReadEntrycount = static_cast<NumType>(tempU64); @@ -689,7 +578,7 @@ // Read map rpos if map exists. if (GetFlag(RwfRwHasMap)) { - ReadAdaptive1248(iStrm, tempU64); + mpt::IO::ReadAdaptiveInt64LE(iStrm, tempU64); if(tempU64 > static_cast<uint64>(std::numeric_limits<Offtype>::max())) { AddReadNote(SNR_INSUFFICIENT_STREAM_OFFTYPE); return; } } @@ -735,7 +624,7 @@ // Read ID. uint16 nIdsize = m_nIdbytes; if(nIdsize == IdSizeVariable) //Variablesize ID - ReadAdaptive12(iStrm, nIdsize); + mpt::IO::ReadAdaptiveInt16LE(iStrm, nIdsize); const size_t nOldEnd = m_Idarray.size(); if (nIdsize > 0 && (Util::MaxValueOfType(nOldEnd) - nOldEnd >= nIdsize)) { @@ -749,7 +638,7 @@ if(GetFlag(RwfRMapHasStartpos)) { uint64 tempU64; - ReadAdaptive1248(iStrm, tempU64); + mpt::IO::ReadAdaptiveInt64LE(iStrm, tempU64); if(tempU64 > static_cast<uint64>(std::numeric_limits<Offtype>::max())) { AddReadNote(SNR_INSUFFICIENT_STREAM_OFFTYPE); return; } mapData[i].rposStart = static_cast<RposType>(tempU64); @@ -761,7 +650,7 @@ else if(GetFlag(RwfRMapHasSize)) // Map has datasize field. { uint64 tempU64; - ReadAdaptive1248(iStrm, tempU64); + mpt::IO::ReadAdaptiveInt64LE(iStrm, tempU64); if(tempU64 > static_cast<uint64>(std::numeric_limits<Offtype>::max())) { AddReadNote(SNR_INSUFFICIENT_STREAM_OFFTYPE); return; } mapData[i].nSize = static_cast<DataSize>(tempU64); @@ -775,7 +664,7 @@ if(GetFlag(RwfRMapHasDesc)) //Map has entrydescriptions? { uint16 size = 0; - ReadAdaptive12(iStrm, size); + mpt::IO::ReadAdaptiveInt16LE(iStrm, size); if(GetFlag(RwfRTwoBytesDescChar)) iStrm.ignore(size * 2); else @@ -853,13 +742,18 @@ // Write entry count. oStrm.seekp(m_posEntrycount); - Binarywrite<size_t>(oStrm, (m_nCounter << 2) | 1, 2); + // Write a fixed size=2 Adaptive64LE because space for this value has already been reserved berforehand. + mpt::IO::WriteAdaptiveInt64LE(oStrm, m_nCounter, 2, 2); + if (GetFlag(RwfRwHasMap)) { // Write map start position. oStrm.seekp(m_posMapPosField); const uint64 rposMap = posMapStart - m_posStart; - Binarywrite<uint64>(oStrm, rposMap << 2 | 3); + + // Write a fixed size=8 Adaptive64LE because space for this value has already been reserved berforehand. + mpt::IO::WriteAdaptiveInt64LE(oStrm, rposMap, 8, 8); + } // Seek to end. Modified: trunk/OpenMPT/common/serialization_utils.h =================================================================== --- trunk/OpenMPT/common/serialization_utils.h 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/common/serialization_utils.h 2014-09-12 10:22:15 UTC (rev 4279) @@ -10,6 +10,10 @@ #pragma once +#include "../common/typedefs.h" +#include "../common/mptIO.h" +#include "../common/Endianness.h" + #include <algorithm> #include <bitset> #include <istream> @@ -24,8 +28,6 @@ #include <cstring> -#include "../common/typedefs.h" - OPENMPT_NAMESPACE_BEGIN namespace srlztn //SeRiaLiZaTioN @@ -73,15 +75,6 @@ }; -void ReadAdaptive12(std::istream& iStrm, uint16& val); -void ReadAdaptive1234(std::istream& iStrm, uint32& val); -void ReadAdaptive1248(std::istream& iStrm, uint64& val); - -void WriteAdaptive12(std::ostream& oStrm, const uint16 num); -void WriteAdaptive1234(std::ostream& oStrm, const uint32 num); -void WriteAdaptive1248(std::ostream& oStrm, const uint64& val); - - enum { IdSizeVariable = uint16_max, @@ -126,25 +119,15 @@ inline void Binarywrite(std::ostream& oStrm, const T& data) //--------------------------------------------------------- { - char b[sizeof(T)]; - std::memcpy(b, &data, sizeof(T)); - #ifdef MPT_PLATFORM_BIG_ENDIAN - std::reverse(b, b+sizeof(T)); - #endif - oStrm.write(b, sizeof(T)); + mpt::IO::WriteIntLE(oStrm, data); } -//Write only given number of bytes from the beginning. -template<class T> -inline void Binarywrite(std::ostream& oStrm, const T& data, const std::size_t bytecount) -//-------------------------------------------------------------------------------------- +template<> +inline void Binarywrite(std::ostream& oStrm, const float& data) +//------------------------------------------------------------- { - char b[sizeof(T)]; - std::memcpy(b, &data, sizeof(T)); - #ifdef MPT_PLATFORM_BIG_ENDIAN - std::reverse(b, b+sizeof(T)); - #endif - oStrm.write(b, std::min(bytecount, sizeof(T))); + IEEE754binary32LE tmp = IEEE754binary32LE(data); + mpt::IO::Write(oStrm, tmp); } template <class T> @@ -170,29 +153,24 @@ inline void Binaryread(std::istream& iStrm, T& data) //-------------------------------------------------- { - char b[sizeof(T)]; - iStrm.read(b, sizeof(T)); - #ifdef MPT_PLATFORM_BIG_ENDIAN - std::reverse(b, b+sizeof(T)); - #endif - std::memcpy(&data, b, sizeof(T)); + mpt::IO::ReadIntLE(iStrm, data); } +template<> +inline void Binaryread(std::istream& iStrm, float& data) +//------------------------------------------------------ +{ + IEEE754binary32LE tmp = IEEE754binary32LE(0.0f); + mpt::IO::Read(iStrm, tmp); + data = tmp; +} + //Read only given number of bytes to the beginning of data; data bytes are memset to 0 before reading. template <class T> inline void Binaryread(std::istream& iStrm, T& data, const Offtype bytecount) //--------------------------------------------------------------------------- { - #ifdef HAS_TYPE_TRAITS - static_assert(std::is_trivial<T>::value == true, ""); - #endif - char b[sizeof(T)]; - std::memset(b, 0, sizeof(T)); - iStrm.read(b, std::min(static_cast<std::size_t>(bytecount), sizeof(T))); - #ifdef MPT_PLATFORM_BIG_ENDIAN - std::reverse(b, b+sizeof(T)); - #endif - std::memcpy(&data, b, sizeof(T)); + mpt::IO::ReadBinaryTruncatedLE(iStrm, data, static_cast<std::size_t>(bytecount)); } @@ -209,36 +187,6 @@ Binaryread(iStrm, data, nSize); } -// Read specialization for float. If data size is 8, read double and assign it to given float. -template <> -inline void ReadItem<float>(std::istream& iStrm, float& f, const DataSize nSize) -//------------------------------------------------------------------------------ -{ - if (nSize == 8) - { - double d; - Binaryread(iStrm, d); - f = static_cast<float>(d); - } - else - Binaryread(iStrm, f); -} - -// Read specialization for double. If data size is 4, read float and assign it to given double. -template <> -inline void ReadItem<double>(std::istream& iStrm, double& d, const DataSize nSize) -//-------------------------------------------------------------------------------- -{ - if (nSize == 4) - { - float f; - Binaryread(iStrm, f); - d = f; - } - else - Binaryread(iStrm, d); -} - void ReadItemString(std::istream& iStrm, std::string& str, const DataSize); template <> @@ -259,6 +207,13 @@ public: + SsbStatus GetStatus() const + { + return m_Status; + } + +protected: + // When writing, returns the number of entries written. // When reading, returns the number of entries read not including unrecognized entries. NumType GetCounter() const {return m_nCounter;} @@ -266,13 +221,6 @@ void SetFlag(Rwf flag, bool val) {m_Flags.set(flag, val);} bool GetFlag(Rwf flag) const {return m_Flags[flag];} - SsbStatus GetStatus() const - { - return m_Status; - } - -protected: - // Write given string to log if log func is defined. void AddToLog(const char *psz); Modified: trunk/OpenMPT/common/stdafx.h =================================================================== --- trunk/OpenMPT/common/stdafx.h 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/common/stdafx.h 2014-09-12 10:22:15 UTC (rev 4279) @@ -47,6 +47,7 @@ #include "../common/typedefs.h" // <memory> // <new> +// <type_traits> // if available // <cstdarg> // <cstdint> // <stdint.h> Modified: trunk/OpenMPT/common/typedefs.h =================================================================== --- trunk/OpenMPT/common/typedefs.h 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/common/typedefs.h 2014-09-12 10:22:15 UTC (rev 4279) @@ -421,6 +421,43 @@ +OPENMPT_NAMESPACE_END +#if defined(HAS_TYPE_TRAITS) +#include <type_traits> +#endif +OPENMPT_NAMESPACE_BEGIN + +namespace mpt { + +#if defined(HAS_TYPE_TRAITS) + +typedef std::true_type true_type; +typedef std::false_type false_type; + +#else // !HAS_TYPE_TRAITS + +struct true_type { + typedef true_type type; + typedef bool value_type; + static const value_type value = true; + operator value_type () const { return value; } + value_type operator () () const { return value; } +}; + +struct false_type { + typedef true_type type; + typedef bool value_type; + static const value_type value = false; + operator value_type () const { return value; } + value_type operator () () const { return value; } +}; + +#endif // HAS_TYPE_TRAITS + +} // namespace mpt + + + #if !defined(MPT_USE_WINDOWS_H) // openmpt assumes these type have exact WIN32 semantics @@ -438,7 +475,55 @@ +namespace mpt { +// Tell which types are safe to binary write into files. +// By default, no types are safe. +// When a safe type gets defined, +// also specialize this template so that IO functions will work. +template <typename T> struct is_binary_safe : public mpt::false_type { }; + +// Specialization for byte types. +template <> struct is_binary_safe<char> : public mpt::true_type { }; +template <> struct is_binary_safe<uint8> : public mpt::true_type { }; +template <> struct is_binary_safe<int8> : public mpt::true_type { }; + +template <typename T> +struct GetRawBytesFunctor +{ + inline const uint8 * operator () (const T & v) const + { + STATIC_ASSERT(mpt::is_binary_safe<T>::value); + return reinterpret_cast<const uint8 *>(&v); + } + inline uint8 * operator () (T & v) const + { + STATIC_ASSERT(mpt::is_binary_safe<T>::value); + return reinterpret_cast<uint8 *>(&v); + } +}; + +// In order to be able to partially specialize it, +// GetRawBytes is implemented via a class template. +// Do not overload or specialize GetRawBytes directly. +// Using a wrapper (by default just around a cast to const uint8 *), +// allows for implementing raw memroy access +// via on-demand generating a cached serialized representation. +template <typename T> inline const uint8 * GetRawBytes(const T & v) +{ + STATIC_ASSERT(mpt::is_binary_safe<T>::value); + return mpt::GetRawBytesFunctor<T>()(v); +} +template <typename T> inline uint8 * GetRawBytes(T & v) +{ + STATIC_ASSERT(mpt::is_binary_safe<T>::value); + return mpt::GetRawBytesFunctor<T>()(v); +} + +} // namespace mpt + + + #if MPT_COMPILER_GCC || MPT_COMPILER_CLANG #define MPT_PRINTF_FUNC(formatstringindex,varargsindex) __attribute__((format(printf, formatstringindex, varargsindex))) #else Modified: trunk/OpenMPT/libopenmpt/libopenmpt.vcxproj =================================================================== --- trunk/OpenMPT/libopenmpt/libopenmpt.vcxproj 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/libopenmpt/libopenmpt.vcxproj 2014-09-12 10:22:15 UTC (rev 4279) @@ -185,6 +185,7 @@ <ClInclude Include="..\common\misc_util.h" /> <ClInclude Include="..\common\mptAtomic.h" /> <ClInclude Include="..\common\mptFstream.h" /> + <ClInclude Include="..\common\mptIO.h" /> <ClInclude Include="..\common\mptPathString.h" /> <ClInclude Include="..\common\mptString.h" /> <ClInclude Include="..\common\mutex.h" /> Modified: trunk/OpenMPT/libopenmpt/libopenmpt.vcxproj.filters =================================================================== --- trunk/OpenMPT/libopenmpt/libopenmpt.vcxproj.filters 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/libopenmpt/libopenmpt.vcxproj.filters 2014-09-12 10:22:15 UTC (rev 4279) @@ -263,6 +263,9 @@ <ClInclude Include="..\common\mptAtomic.h"> <Filter>Header Files\common</Filter> </ClInclude> + <ClInclude Include="..\common\mptIO.h"> + <Filter>Header Files\common</Filter> + </ClInclude> </ItemGroup> <ItemGroup> <ClCompile Include="..\common\AudioCriticalSection.cpp"> Modified: trunk/OpenMPT/libopenmpt/libopenmptDLL.vcxproj =================================================================== --- trunk/OpenMPT/libopenmpt/libopenmptDLL.vcxproj 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/libopenmpt/libopenmptDLL.vcxproj 2014-09-12 10:22:15 UTC (rev 4279) @@ -193,6 +193,7 @@ <ClInclude Include="..\common\misc_util.h" /> <ClInclude Include="..\common\mptAtomic.h" /> <ClInclude Include="..\common\mptFstream.h" /> + <ClInclude Include="..\common\mptIO.h" /> <ClInclude Include="..\common\mptPathString.h" /> <ClInclude Include="..\common\mptString.h" /> <ClInclude Include="..\common\mutex.h" /> Modified: trunk/OpenMPT/libopenmpt/libopenmptDLL.vcxproj.filters =================================================================== --- trunk/OpenMPT/libopenmpt/libopenmptDLL.vcxproj.filters 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/libopenmpt/libopenmptDLL.vcxproj.filters 2014-09-12 10:22:15 UTC (rev 4279) @@ -269,6 +269,9 @@ <ClInclude Include="..\common\mptAtomic.h"> <Filter>Header Files\common</Filter> </ClInclude> + <ClInclude Include="..\common\mptIO.h"> + <Filter>Header Files\common</Filter> + </ClInclude> </ItemGroup> <ItemGroup> <ClCompile Include="..\common\AudioCriticalSection.cpp"> Modified: trunk/OpenMPT/libopenmpt/libopenmpt_test.vcxproj =================================================================== --- trunk/OpenMPT/libopenmpt/libopenmpt_test.vcxproj 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/libopenmpt/libopenmpt_test.vcxproj 2014-09-12 10:22:15 UTC (rev 4279) @@ -189,6 +189,7 @@ <ClInclude Include="..\common\misc_util.h" /> <ClInclude Include="..\common\mptAtomic.h" /> <ClInclude Include="..\common\mptFstream.h" /> + <ClInclude Include="..\common\mptIO.h" /> <ClInclude Include="..\common\mptPathString.h" /> <ClInclude Include="..\common\mptString.h" /> <ClInclude Include="..\common\mutex.h" /> Modified: trunk/OpenMPT/libopenmpt/libopenmpt_test.vcxproj.filters =================================================================== --- trunk/OpenMPT/libopenmpt/libopenmpt_test.vcxproj.filters 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/libopenmpt/libopenmpt_test.vcxproj.filters 2014-09-12 10:22:15 UTC (rev 4279) @@ -263,6 +263,9 @@ <ClInclude Include="..\common\mptAtomic.h"> <Filter>Header Files\common</Filter> </ClInclude> + <ClInclude Include="..\common\mptIO.h"> + <Filter>Header Files\common</Filter> + </ClInclude> </ItemGroup> <ItemGroup> <ClCompile Include="..\common\AudioCriticalSection.cpp"> Modified: trunk/OpenMPT/mptrack/mptrack_08.vcproj =================================================================== --- trunk/OpenMPT/mptrack/mptrack_08.vcproj 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/mptrack/mptrack_08.vcproj 2014-09-12 10:22:15 UTC (rev 4279) @@ -1346,6 +1346,10 @@ > </File> <File + RelativePath="..\common\mptIO.h" + > + </File> + <File RelativePath="..\common\mptPathString.h" > </File> Modified: trunk/OpenMPT/mptrack/mptrack_10.vcxproj =================================================================== --- trunk/OpenMPT/mptrack/mptrack_10.vcxproj 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/mptrack/mptrack_10.vcxproj 2014-09-12 10:22:15 UTC (rev 4279) @@ -649,6 +649,7 @@ <ClInclude Include="..\common\misc_util.h" /> <ClInclude Include="..\common\mptAtomic.h" /> <ClInclude Include="..\common\mptFstream.h" /> + <ClInclude Include="..\common\mptIO.h" /> <ClInclude Include="..\common\mptPathString.h" /> <ClInclude Include="..\common\mptString.h" /> <ClInclude Include="..\common\mutex.h" /> Modified: trunk/OpenMPT/mptrack/mptrack_10.vcxproj.filters =================================================================== --- trunk/OpenMPT/mptrack/mptrack_10.vcxproj.filters 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/mptrack/mptrack_10.vcxproj.filters 2014-09-12 10:22:15 UTC (rev 4279) @@ -1023,6 +1023,9 @@ <ClInclude Include="..\common\WriteMemoryDump.h"> <Filter>Header Files\common</Filter> </ClInclude> + <ClInclude Include="..\common\mptIO.h"> + <Filter>Header Files\common</Filter> + </ClInclude> </ItemGroup> <ItemGroup> <None Include="res\bitmap1.bmp"> Modified: trunk/OpenMPT/soundlib/Load_it.cpp =================================================================== --- trunk/OpenMPT/soundlib/Load_it.cpp 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/soundlib/Load_it.cpp 2014-09-12 10:22:15 UTC (rev 4279) @@ -16,6 +16,7 @@ #include "../mptrack/moddoc.h" #include "../mptrack/TrackerSettings.h" #endif +#include "../common/mptIO.h" #include "../common/serialization_utils.h" #include "../common/mptFstream.h" #include <sstream> @@ -109,7 +110,7 @@ else //Case: Using original IT tuning. srlztn::StringToBinaryStream<uint8>(oStrm, "->MPT_ORIGINAL_IT<-"); - srlztn::Binarywrite<uint16>(oStrm, iter->second); + mpt::IO::WriteIntLE<uint16>(oStrm, iter->second); } //Writing tuning data for instruments. @@ -121,7 +122,7 @@ sf.AddToLog("Error: 210807_1"); return; } - srlztn::Binarywrite(oStrm, iter->second); + mpt::IO::WriteIntLE<uint16>(oStrm, iter->second); } } } @@ -1269,9 +1270,18 @@ itHeader.ConvertEndianness(); Order.WriteAsByte(f, itHeader.ordnum); - if(itHeader.insnum) fwrite(&inspos[0], 4, itHeader.insnum, f); - if(itHeader.smpnum) fwrite(&smppos[0], 4, itHeader.smpnum, f); - if(itHeader.patnum) fwrite(&patpos[0], 4, itHeader.patnum, f); + for(uint16 i = 0; i < itHeader.insnum; ++i) + { + mpt::IO::WriteIntLE<uint32>(f, inspos[i]); + } + for(uint16 i = 0; i < itHeader.smpnum; ++i) + { + mpt::IO::WriteIntLE<uint32>(f, smppos[i]); + } + for(uint16 i = 0; i < itHeader.patnum; ++i) + { + mpt::IO::WriteIntLE<uint32>(f, patpos[i]); + } // Writing edit history information SaveITEditHistory(this, f); @@ -1288,8 +1298,7 @@ char magic[4]; memcpy(magic, "PNAM", 4); fwrite(magic, 4, 1, f); - uint32 d = numNamedPats * MAX_PATTERNNAME; - fwrite(&d, 4, 1, f); + mpt::IO::WriteIntLE<uint32>(f, numNamedPats * MAX_PATTERNNAME); for(PATTERNINDEX nPat = 0; nPat < numNamedPats; nPat++) { @@ -1306,7 +1315,7 @@ char magic[4]; memcpy(magic, "CNAM", 4); fwrite(magic, 4, 1, f); - fwrite(&dwChnNamLen, 1, 4, f); + mpt::IO::WriteIntLE<uint32>(f, dwChnNamLen); UINT nChnNames = dwChnNamLen / MAX_CHANNELNAME; for(UINT inam = 0; inam < nChnNames; inam++) { @@ -1590,9 +1599,18 @@ // Updating offsets fseek(f, dwHdrPos, SEEK_SET); - if(itHeader.insnum) fwrite(&inspos[0], 4, itHeader.insnum, f); - if(itHeader.smpnum) fwrite(&smppos[0], 4, itHeader.smpnum, f); - if(itHeader.patnum) fwrite(&patpos[0], 4, itHeader.patnum, f); + for(uint16 i = 0; i < itHeader.insnum; ++i) + { + mpt::IO::WriteIntLE<uint32>(f, inspos[i]); + } + for(uint16 i = 0; i < itHeader.smpnum; ++i) + { + mpt::IO::WriteIntLE<uint32>(f, smppos[i]); + } + for(uint16 i = 0; i < itHeader.patnum; ++i) + { + mpt::IO::WriteIntLE<uint32>(f, patpos[i]); + } if(GetType() == MOD_TYPE_IT) { @@ -1656,7 +1674,7 @@ fout.clear(); success = false; } - fout.write(reinterpret_cast<const char*>(&MPTStartPos), sizeof(MPTStartPos)); + mpt::IO::WriteIntLE<uint32>(fout, MPTStartPos); fout.flush(); #if MPT_COMPILER_MSVC @@ -1708,7 +1726,7 @@ } // rewbs.modularPlugData - DWORD MPTxPlugDataSize = 4 + (sizeof(m_MixPlugins[i].fDryRatio)) + // 4 for ID and size of dryRatio + DWORD MPTxPlugDataSize = 4 + (sizeof(m_MixPlugins[i].fDryRatio)) + //4 for ID and size of dryRatio 4 + (sizeof(m_MixPlugins[i].defaultProgram)) + // rewbs.plugDefaultProgram 4 + 3 * sizeof(int32); // Editor data // for each extra entity, add 4 for ID, plus size of entity, plus optionally 4 for size of entity. @@ -1725,30 +1743,31 @@ fwrite(id, 1, 4, f); // write plugin size: - fwrite(&nPluginSize, 1, 4, f); - fwrite(&plugin.Info, 1, sizeof(SNDMIXPLUGININFO), f); - fwrite(&m_MixPlugins[i].nPluginDataSize, 1, 4, f); + mpt::IO::WriteIntLE<uint32>(f, nPluginSize); + SNDMIXPLUGININFO tmpPluginInfo = m_MixPlugins[i].Info; + tmpPluginInfo.ConvertEndianness(); + fwrite(&tmpPluginInfo, 1, sizeof(SNDMIXPLUGININFO), f); + mpt::IO::WriteIntLE<uint32>(f, m_MixPlugins[i].nPluginDataSize); if(m_MixPlugins[i].pPluginData) { fwrite(m_MixPlugins[i].pPluginData, 1, m_MixPlugins[i].nPluginDataSize, f); } - fwrite(&MPTxPlugDataSize, 1, 4, f); + mpt::IO::WriteIntLE<uint32>(f, MPTxPlugDataSize); // Dry/Wet ratio memcpy(id, "DWRT", 4); fwrite(id, 1, 4, f); - uint32 ratio = IEEE754binary32LE(m_MixPlugins[i].fDryRatio).GetInt32(); - fwrite(&ratio, 1, sizeof(uint32), f); + // DWRT chunk does not include a size, so better make sure we always write 4 bytes here. + STATIC_ASSERT(sizeof(IEEE754binary32LE) == 4); + mpt::IO::Write(f, IEEE754binary32LE(m_MixPlugins[i].fDryRatio)); // Default program memcpy(id, "PROG", 4); fwrite(id, 1, 4, f); // PROG chunk does not include a size, so better make sure we always write 4 bytes here. STATIC_ASSERT(sizeof(m_MixPlugins[i].defaultProgram) == sizeof(int32)); - int32 prog = m_MixPlugins[i].defaultProgram; - SwapBytesLE(prog); - fwrite(&prog, 1, sizeof(int32), f); + mpt::IO::WriteIntLE<int32>(f, m_MixPlugins[i].defaultProgram); // Editor window parameters memcpy(id, "EWND", 4); @@ -1783,9 +1802,11 @@ { memcpy(id, "CHFX", 4); fwrite(id, 1, 4, f); - nPluginSize = nChInfo * 4; - fwrite(&nPluginSize, 1, 4, f); - fwrite(chinfo, 1, nPluginSize, f); + mpt::IO::WriteIntLE<uint32>(f, nChInfo * 4); + for(uint32 i = 0; i < nChInfo; ++i) + { + mpt::IO::WriteIntLE<uint32>(f, chinfo[i]); + } } nTotalSize += nChInfo * 4 + 8; } @@ -1912,7 +1933,7 @@ //--------------------------------------------------------------------------------- { uint32 code = MAGIC4BE('M','P','T','X'); // write extension header code - fwrite(&code, 1, sizeof(uint32), f); + mpt::IO::WriteIntLE<uint32>(f, code); if (nInstruments == 0) return; @@ -1974,8 +1995,8 @@ void CSoundFile::WriteInstrumentPropertyForAllInstruments(uint32 code, int16 size, FILE* f, UINT nInstruments) const //------------------------------------------------------------------------------------------------------------------ { - fwrite(&code, 1, sizeof(uint32), f); //write code - fwrite(&size, 1, sizeof(int16), f); //write size + mpt::IO::WriteIntLE<uint32>(f, code); //write code + mpt::IO::WriteIntLE<int16>(f, size); //write size for(UINT nins=1; nins<=nInstruments; nins++) //for all instruments... { if (Instruments[nins]) @@ -1998,16 +2019,16 @@ #define WRITEMODULARHEADER(c1, c2, c3, c4, fsize) \ { \ - const uint32 code = SwapBytesReturnLE(MAGIC4BE(c1, c2, c3, c4)); \ - fwrite(&code, 1, sizeof(code), f); \ - ASSERT(fsize <= uint16_max); \ - const uint16 size = SwapBytesReturnLE(static_cast<uint16>(fsize)); \ - fwrite(&size, 1, sizeof(size), f); \ + const uint32 code = MAGIC4BE(c1, c2, c3, c4); \ + mpt::IO::WriteIntLE<uint32>(f, code); \ + ASSERT(fsize <= uint16_max); \ + const uint16 size = fsize; \ + mpt::IO::WriteIntLE<uint16>(f, size); \ } #define WRITEMODULAR(c1, c2, c3, c4, field) \ { \ WRITEMODULARHEADER(c1, c2, c3, c4, sizeof(field)) \ - fwrite(&(field), 1, sizeof(field), f); \ + mpt::IO::WriteIntLE(f, field); \ } if(m_nDefaultTempo > 255) @@ -2068,7 +2089,7 @@ //Additional flags for XM/IT/MPTM if(m_ModFlags) { - WRITEMODULAR('M','S','F','.', m_ModFlags); + WRITEMODULAR('M','S','F','.', m_ModFlags.GetRaw()); } #ifdef MODPLUG_TRACKER @@ -2287,8 +2308,7 @@ //write modular data's total size long curPos = ftell(f); // remember current pos fseek(f, sizePos, SEEK_SET); // go back to sizePos - SwapBytesLE(modularInstSize); - fwrite(&modularInstSize, 1, sizeof(modularInstSize), f); // write data + mpt::IO::WriteIntLE<uint32>(f, modularInstSize); // write data fseek(f, curPos, SEEK_SET); // go back to where we were. // Compute the size that we just wasted. Modified: trunk/OpenMPT/soundlib/ModSequence.cpp =================================================================== --- trunk/OpenMPT/soundlib/ModSequence.cpp 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/soundlib/ModSequence.cpp 2014-09-12 10:22:15 UTC (rev 4279) @@ -778,7 +778,7 @@ //----------------------------------------------------------------------------- { uint16 size; - srlztn::Binaryread<uint16>(iStrm, size); + mpt::IO::ReadIntLE<uint16>(iStrm, size); if(size > ModSpecs::mptm.ordersMax) { seq.m_sndFile.AddToLog(mpt::String::Print(str_SequenceTruncationNote, size, ModSpecs::mptm.ordersMax)); @@ -791,7 +791,7 @@ for(size_t i = 0; i < size; i++) { uint16 temp; - srlztn::Binaryread<uint16>(iStrm, temp); + mpt::IO::ReadIntLE<uint16>(iStrm, temp); seq[i] = temp; } } @@ -801,12 +801,12 @@ //---------------------------------------------------------------------- { const uint16 size = seq.GetLength(); - srlztn::Binarywrite<uint16>(oStrm, size); + mpt::IO::WriteIntLE<uint16>(oStrm, size); const ModSequenceSet::const_iterator endIter = seq.end(); for(ModSequenceSet::const_iterator citer = seq.begin(); citer != endIter; citer++) { const uint16 temp = static_cast<uint16>(*citer); - srlztn::Binarywrite<uint16>(oStrm, temp); + mpt::IO::WriteIntLE<uint16>(oStrm, temp); } } Modified: trunk/OpenMPT/soundlib/Sndfile.cpp =================================================================== --- trunk/OpenMPT/soundlib/Sndfile.cpp 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/soundlib/Sndfile.cpp 2014-09-12 10:22:15 UTC (rev 4279) @@ -18,6 +18,7 @@ #endif // MODPLUG_TRACKER #include "../common/version.h" #include "../common/AudioCriticalSection.h" +#include "../common/mptIO.h" #include "../common/serialization_utils.h" #include "Sndfile.h" #include "tuningcollection.h" @@ -189,8 +190,8 @@ fsize = sizeof( type );\ if(only_this_code == Util::MaxValueOfType(only_this_code)) \ { \ - fwrite(& fcode , 1 , sizeof( uint32 ) , file);\ - fwrite(& fsize , 1 , sizeof( int16 ) , file);\ + mpt::IO::WriteIntLE<uint32>(file, fcode); \ + mpt::IO::WriteIntLE<int16>(file, fsize); \ } else if(only_this_code == fcode)\ { \ ASSERT(fixedsize == fsize); \ @@ -212,8 +213,8 @@ fsize = sizeof( type );\ if(only_this_code == Util::MaxValueOfType(only_this_code)) \ { \ - fwrite(& fcode , 1 , sizeof( uint32 ) , file);\ - fwrite(& fsize , 1 , sizeof( int16 ) , file);\ + mpt::IO::WriteIntLE<uint32>(file, fcode); \ + mpt::IO::WriteIntLE<int16>(file, fsize); \ type tmp = (type)(input-> name ); \ tmp = SwapBytesReturnLE(tmp); \ fwrite(&tmp , 1 , fsize , file); \ @@ -247,8 +248,8 @@ fsize = sizeof( type ) * arraysize;\ if(only_this_code == Util::MaxValueOfType(only_this_code)) \ { \ - fwrite(& fcode , 1 , sizeof( uint32 ) , file);\ - fwrite(& fsize , 1 , sizeof( int16 ) , file);\ + mpt::IO::WriteIntLE<uint32>(file, fcode); \ + mpt::IO::WriteIntLE<int16>(file, fsize); \ } else if(only_this_code == fcode)\ { \ /* ASSERT(fixedsize <= fsize); */ \ @@ -311,8 +312,8 @@ fsize = sizeof(dwFlags); if(!only_this_code) { - fwrite(&fcode, 1, sizeof(int32), file); - fwrite(&fsize, 1, sizeof(int16), file); + mpt::IO::WriteIntLE<int32>(file, fcode); + mpt::IO::WriteIntLE<int16>(file, fsize); } dwFlags = SwapBytesReturnLE(dwFlags); fwrite(&dwFlags, 1, fsize, file); Modified: trunk/OpenMPT/soundlib/pattern.cpp =================================================================== --- trunk/OpenMPT/soundlib/pattern.cpp 2014-09-12 01:06:27 UTC (rev 4278) +++ trunk/OpenMPT/soundlib/pattern.cpp 2014-09-12 10:22:15 UTC (rev 4279) @@ -532,8 +532,6 @@ return mask; } -using srlztn::Binarywrite; -using srlztn::Binaryread; // Writes pattern data. Adapted from SaveIT. void WriteData(std::ostream& oStrm, const CPattern& pat) @@ -560,21 +558,21 @@ if(diffmask != 0) chval |= IT_bitmask_patternChanEnabled_c; ... [truncated message content] |