UPX and TLS

Max
2010-03-23
2013-05-23
1 2 > >> (Page 1 of 2)
  • Max
    Max
    2010-03-23

    Dear UPX Devel,

    Recently MinGW and GCC team have implemented full support for TLS in the PE/COFF created with GCC (MinGW) As a result
    UPX is no longer usable on executables. Hereater the error I got with an exe created using latest MinGW runtime

                           Ultimate Packer for eXecutables
                              Copyright (C) 1996 - 2009
    UPX 3.04w       Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 27th 2009

            File size         Ratio      Format      Name
       -------------   ----   -------   -------
    upx: obj\JenkinsTraubDbg.exe: CantPackException: TLS callbacks are not supported

    Packed 1 file: 0 ok, 1 error.

    Process returned 1 (0x1)   execution time : 0.234 s
    Press any key to continue.

    See also

    https://sourceforge.net/mailarchive/forum.php?thread_name=90baa01f1003200523q5c41e95bx938c7baeeba82560%40mail.gmail.com&forum_name=mingw-dvlpr

    Is it possible to fix UPX?

    Thx

    Regards

    Max

     
  • At the time PE support was added, I had not found any documentation on TLS callbacks. Can you provide me with some now?

    Laszlo

     
  • Thanks for the documentation. I still has a question here: are these tls callbacks called/accessed in *any* way by the OS loader before the call of the entry point of the executable? If the answer is yes, then there will be no support for TLS callbacks in UPX. Otherwise it's just a matter of removing the error message.

    I'm asking this, because the TLS index field is accessed (written) by the loader before the decompression code is started. But in that case I was able to add a workaround…

    L

     
  • Charles Wilson
    Charles Wilson
    2010-04-09

    As far as I can tell, it's all handled by the normal "C" windows entry points (even for other mingw-supported languages like Fortran, AFAICT). For console apps, that's mainCRTStartup; for GUI apps, it's WinMainCRTStartup. Both are provided by mingw's crt1.c file, and actually, both quickly hand off to the same core function __mingw_CRTStartup() - which is where the TLS initialization is done.  For DLLs, it's all handled by the windows standard DLL entry point DllMainCRTStartup() (provided by mingw's dllcrt1.c), and TLS initialization is handed by the PROCESS_ATTACH/THREAD_ATTACH cases.

    Note that the msdn documentation talks about custom entry points; however, with mingw-compiled DLLs and apps, control is always passed first to a specific runtime initializer first - as described above - before user initialization routines such as DllMain() are called. The above discussion concerns these mingw-compiler-provided initialization routines, since they always precede any user-supplied ones.

    For cygwin processes, everything is handled by the entry point for DLLs (__cygwin_dll_entry@12), or by the standard entry point for windows executables WinMainCRTStartup.
    http://cygwin.com/ml/cygwin-patches/2010-q1/msg00057.html
    (Note that cygwin has its own TLS implementation support, so the current discussion about recent mingw changes has no meaning with regards to cygwin executables and DLLs).

     
  • TTwrs
    TTwrs
    2010-04-12

    This is one take on the issue that I ran across.  Their feeling is that UPX would be more robust if it could deal with TLS callbacks.  As an end user I can say that there definitely IS a problem. ;>)  All the sudden I couldn't pack new versions of programs that I was able to pack previously… :(

    http://permalink.gmane.org/gmane.comp.gnu.mingw.devel/3737

    Hopefully *someone* can find a solution .. I love UPX! ;)

     
  • My question is still: are these tls callbacks called/accessed in *any* way by the OS *loader* before the call of the entry point of the executable?

    L

     
  • Stefan Widmann
    Stefan Widmann
    2010-04-13

    Hello László,

    yes, the TLS callbacks are called by the PE loader before the entry point is executed.

    From PECOFF v8.1:

    typedef VOID
    (NTAPI *PIMAGE_TLS_CALLBACK) (
        PVOID DllHandle,
        DWORD Reason,
        PVOID Reserved
        );

    All callback are called once during process creation with Reason = DLL_PROCESS_ATTACH (and on every thread creation, too, but that is not our focus).

    There are several ways to handle callbacks in packers, protectors or crypters:

    The easiest way is to remove the pointer to the callback chain (TLS + 0x0C) and restore it from within the stub.

    This did function well, but Vista started saving the pointer during process creation and thus does not recognize any

    change afterwards.

    PECompact moves all callbacks to the relocated TLS, setting the first callback to zero. During decompression, all

    callbacks are pushed forward one dword.

    This approach does function well for all operating systems, including Vista and Win7.

    Please note, that PECOFF explicitly states, that TLS callbacks can be add or removed dynamically during runtime.

    After restoring the TLS callbacks in whatever way you've chosen, you emulate the pre-entrypoint call by the PE

    loader by calling every single callback with Reason = DLL_PROCESS_ATTACH as written above.

    Please see the mail I've wrote you a few days ago.

    Kind regards,

    Stefan Widmann

     
  • Hi Stefan,

    Thanks for the info. I think I now have everything to start implementing this support.

    L

     
  • Stefan Widmann
    Stefan Widmann
    2010-04-14

    Hi László,

    as I wrote you in my E-Mail last week, I started implementing UPX TLS callback support for win32/pe a few weeks ago. It's ready to go for beta testing now. It does not simply copy the callback chain to the TLS section of UPX, since the application might add callbacks dynamically, but not to the new location in the new TLS. Instead, my approach installs a little callback handler in the stub, which calls the original callback chain at the original location.

    Files changed:
    - p_w32pe.cpp
    - i386-win32.pe.S (and .h, of course)

    The complete callback handling takes up 83 bytes in the stub now (I'm sure, there's room to squeeze out a few bytes more), split into 75 bytes for installing the handler, the handler itself and the emulation of the calls with DLL_PROCESS_ATTACH. Two dwords are used to provide a new callback chain to point to the handler.

    Is my contribution welcome? Did you receive my mail?

    Kind regards,

    Stefan

     
  • Hi Stefan,

    Sorry, your first mail was lost. But I've found it in my spam folder, and sent a reply to you.

    bye, Laszlo

     
  • TTwrs
    TTwrs
    2010-05-18

    I finding more and more binaries that can't be UPXed because of the TLS callback error.  Is there any hope for a fix any time soon?  ;>)

     
  • Hi,

    Yes, there is hope. :-) Stefan Widmann is working on the TLS callback support. Hopefully I will be able to integrate his patches into UPX this month.

    bye,Laszlo

     
  • Hi,

    our mercurial repo already contains the TLS callback support code. If anyone wants to test it before the official UPX release, please do, and report back if a problem is found. Thanks.

    L

     
  • TTwrs
    TTwrs
    2010-07-20

    Is there a link for a Win32 binary?  I'll test…

     
  • Max
    Max
    2010-07-20

    >Is there a link for a Win32 binary? I'll test…

    Me to. I have no way (with my current environment) to get the source code from mercurial (and I don't know how mwercurial is working). A testing exe will help….

     
  • Sorry for the delay, but I'm on vacation now. I'll upload a win32 binary early next month.

    thanks,L

     
  • M. B. Huffman
    M. B. Huffman
    2010-08-06

    I compressed the following installations with upx30590.exe:
    * Strawberry Perl 5.12.1 (compiled with GCC 4.4.3)
    * Parrot 2.6.0
    *  Rakudo Star (a.k.a. Perl 6) version 2010.07-47-g9fd5eaa (compiled with GCC 4.4.3)

    No problems so far based on normal daily Perl (5) programs and limited Perl 6 explorations that ran before compression.

    Mike

     
  • TTwrs
    TTwrs
    2010-08-10

    I use UPX along with the freeUPX GUI.  I tried comI tried compressing mplayer version 31878 from…
    https://sourceforge.net/projects/mplayer-win32/files/MPlayer%20and%20MEncoder/revision%2031878/MPlayer-rtm-svn-31878.7z/download

    The following command line was passed to UPX throught the FreeUPX GUI…
    upx.exe -best -all-methods -compress-resource=0 mplayer.exe

    It seemed to end up in an endless loop trying the 'first method' (of four).  The CPU usage went to 95+% (ususal when compressing mplayer), and just stuck there for about 15-minutes at which time I aborted the process before my processor melted. ;)

    I tried it with a couple of other exe files with similar results.

    I then tried using…
    upx.exe -9 -compress-resource=0 mplayer.exe
    … and mplayer compressed just fine to 43% of its original size.

    So, there may be a problem with the "-all-methods" option.  Otherwise, very encouraging.  Thanks.

     
  • Hi ttwrs,

    This is what I get after ~14 minutes on my computer:

       /tmp/upx/src/upx.out  -best -all-methods -compress-resource=0 mplayer.exe
                              Ultimate Packer for eXecutables
                                 Copyright (C) 1996 - 2010
       UPX 3.06b       Markus Oberhumer, Laszlo Molnar & John Reiser    Aug 3rd 2010

               File size         Ratio      Format      Name
          -------------   ----   -------   -------
         17075712 ->   5872128   34.39%    win32/pe     mplayer.exe

       Packed 1 file.

    So it's slow, but it works. Could you try again please?

    Thanks, Laszlo

     
  • TTwrs
    TTwrs
    2010-08-11

    Hi Lazlo,
    These are my notes for latest attempt…

    With fire extinguisher close at hand, I attempted to compress mplayer.exe again…

    pass 1 of 4 took 40 minutes.
    After 60 minutes, pass two hadn't yet completed.
    Needed to use the computer, so had to abort the test.

    .. Later, I tried compressing FreeFileSync.exe V3.9 (with TLS callback problem) with similar results, and aborted after 20 minutes.
    (FreeFileSync_v3.x_setup.exe from http://sourceforge.net/projects/freefilesync/files/, select 'Portable' mode)

    .. Next…
    .. FreeFileSync 3.7 (without TLS callback problem) - UPX 3.06b - 6 minutes.
    .. FreeFileSync 3.7 (without TLS callback problem) - UPX 3.05w - 1 minute, 4 seconds.

    .. So, 'something' isn't right .. nothing has ever taken even close to this much time to compress.
    .. Computer is a WinXP-sp1 2.4 GHz P4.

     
  • Hi,

    Please note that my UPX 3.06b version was built using the open source UCL library which (as it seems) is much slower than the NRV library used in the official UPX builds (which only Markus can produce). So this may be the reason for the performance problems.

    Maybe you could change the command line option "-best" to "-9" for testing, as it seems to be much faster.

    Thanks for your report, L

     
  • TTwrs
    TTwrs
    2010-08-12

    Thanks Lazlo, Mplayer is a bear to compress, which probably makes it a good 'reference'. .. I don't know if the results would be the same with files other than mplayer, but certainly using -9 is almost as good as -best in terms of compression - 43.5% vs. 43.2%, respectively.  Yet it took only 2.75-minutes versus 25-minutes!  Looks like -9 (or even the 'UPX default' of -7) is the way to go.  The real difference comes with using -all-methods, which gave 34.4% versus 43.5%.  Took quite a bit longer (10-minutes) but well worth it, IMO. .. Anyway, it seem like the TLS issue is resolved with 3.06b. Will there be an 'official' release using the NRV library?

     
  • The -all-methods compression ratio improvement is probably caused by that LZMA is also tried. So you could use the -lzma switch instead of -all-methods. That way you can skip all the NRV* methods (which are much faster to decompress).

    The next official UPX release will contain a bit newer version of TLS callback support, and will come with the NRV library. Release date is not yet known.

    L

     
1 2 > >> (Page 1 of 2)