#214 UPX'd binaries segfault (armel) - was ID: 3595786

None
closed-fixed
None
5
2013-06-13
2013-05-06
Zibri
No

The bug is still there somehow.
I compressed a program I can't publicly disclose and the behaviour is this:
it works the first 3-4 times I run it. Then it segfaults from then on.
But if I run the original exe, it doesn't segfault.
The program uses pthreads, ptrace and a lot of memory.

Discussion

  • Zibri

    Zibri - 2013-05-06

    Oh and the cpu is a quadcore ARMv7 Processor rev 3 (v7l)

     
  • John Reiser

    John Reiser - 2013-05-08

    What version of UPX? What operating system and version? Show the results of "readelf --segments ./my_app".

    Run the compressed program under a debugger such as gdb. When the SIGSEGV happens, then print the registers, the instruction stream, and the process address space:
    (gdb) info reg
    (gdb) x/8i $pc-4*4
    (gdb) info proc
    (gdb) shell cat /proc/<PID>/maps

     
  • Zibri

    Zibri - 2013-05-08

    Elf file type is EXEC (Executable file)
    Entry point 0x8c08
    There are 8 program headers, starting at offset 52

    Program Headers:
    Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
    EXIDX 0x003dd0 0x0000bdd0 0x0000bdd0 0x000d8 0x000d8 R 0x4
    PHDR 0x000034 0x00008034 0x00008034 0x00100 0x00100 R E 0x4
    INTERP 0x000134 0x00008134 0x00008134 0x00013 0x00013 R 0x1
    [Requesting program interpreter: /lib/ld-linux.so.3]
    LOAD 0x000000 0x00008000 0x00008000 0x03eac 0x03eac R E 0x8000
    LOAD 0x004000 0x00014000 0x00014000 0x001c5 0x001dc RW 0x8000
    DYNAMIC 0x00400c 0x0001400c 0x0001400c 0x000f8 0x000f8 RW 0x4
    NOTE 0x000148 0x00008148 0x00008148 0x00020 0x00020 R 0x4
    GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x4

    Section to Segment mapping:
    Segment Sections...
    00 .ARM.exidx
    01
    02 .interp
    03 .interp .note.ABI-tag .hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.dyn .rel.plt .init .plt .text .fini .rodata .ARM.extab .ARM.exidx .eh_frame
    04 .init_array .fini_array .jcr .dynamic .got .data .bss
    05 .dynamic
    06 .note.ABI-tag
    07

    # upx --version

    upx 3.08
    UCL data compression library 1.03
    Copyright (C) 1996-2011 Markus Franz Xaver Johannes Oberhumer
    Copyright (C) 1996-2011 Laszlo Molnar
    Copyright (C) 2000-2011 John F. Reiser
    Copyright (C) 2002-2011 Jens Medoch

     
  • John Reiser

    John Reiser - 2013-05-11

    Thank you for posting the output from "readelf --segments". Looking at the Program Headers, I don't see any particular problem. The program looks somewhat small: less than 16KB of .text, less than 500 bytes of .data. Would you please send it to me as an attachment in private e-mail? Try the sourceforge address first (click on my Sender name to get my sourceforge page), then if that does not work use the same name with my com domain "bitwagon".

    I'm still unsure about the execution environment. I see that the CPU is "quadcore ARMv7 Processor rev 3 (v7l)". That's good: has hardware floating point, NEON multimedia instructions, etc. How much RAM? What operating system and version does the program run under? Which C library and version?

    I'm also unsure about the mode of failure. "it works the first 3-4 times I run it. Then it segfaults from then on." Does this happen when the program is invoked serially, with each invocation exiting before the next invocation begins, or are you starting several instances in parallel before any of them finish? Although the program itself is small, it uses "a lot of memory". How much memory (total virtual space per process)? How close are you to running out of RAM or swap space?

    " it segfaults" Please run under a debugger and collect the evidence: register contents, program counter, instruction stream, process address space. For instance:
    (gdb) info reg
    (gdb) x/8i $pc-4*4
    (gdb) info proc
    (gdb) shell cat /proc/<PID>/maps

    Can you run under strace? "strace -f -o strace.out ./my_app <<arguments>>" then attach or send strace.out, or post it to a public "paste" site..

    The point of all of this is that in order to make progress, it is necessary to re-construct the error, either with an actual example, or mentally as a thought experiment. So please send the program, or give enough details for the thought experiment.

    Because the program is so small (16KB), it might be easy to construct a synthetic test case that behaves the same, but which can be distributed freely. Please think about that, too.

     
  • Dennis

    Dennis - 2013-06-05

    I have the same problem with upx'ed arm binaries (on my tablet, armv7 processor as well). I've compiled v3.09 from source on debian (chroot environment on android tablet), used debian 7's stock v3.08 (apt-get), and eventually compiled v3.10 directly on my android device.

    An very simple UPX'ed binary will work fine (a not-too-small hello-world type of program). When you UPX busybox and run it, it will segfault ~ 95% of the time. Sometimes it'll display "Bus error". Just very rarely it'll execute (Seems like a stack problem).

    The problem is still there if you compile in lzma support and use --lzma to compress the binary, but the segfaults are more rare then. (I'd estimate 1 out of 5 runs segfaults on a binary that isn't overly complex/big).

     
    Last edit: Dennis 2013-06-05
  • Dennis

    Dennis - 2013-06-05

    segments UPX'ed busybox
    Elf file type is EXEC (Executable file)
    Entry point 0x9fe68
    There are 2 program headers, starting at offset 52

    Program Headers:
    Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
    LOAD 0x000000 0x00008000 0x00008000 0x98754 0x98754 R E 0x8000
    LOAD 0x005360 0x0011d360 0x0011d360 0x00000 0x00000 RW 0x8000

    Not-UPXed:

    Elf file type is EXEC (Executable file)
    Entry point 0x89e0
    There are 5 program headers, starting at offset 52

    Program Headers:
    Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
    EXIDX 0x10fc00 0x00117c00 0x00117c00 0x00188 0x00188 R 0x4
    LOAD 0x000000 0x00008000 0x00008000 0x10fd8c 0x10fd8c R E 0x8000
    LOAD 0x110000 0x00118000 0x00118000 0x004e6 0x05360 RW 0x8000
    TLS 0x110000 0x00118000 0x00118000 0x00004 0x00010 R 0x4
    GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x4

    Section to Segment mapping:
    Segment Sections...
    00 .ARM.exidx
    01 .init .text .fini .rodata .ARM.extab .ARM.exidx .eh_frame
    02 .tdata .init_array .fini_array .jcr .data.rel.ro .got .data .bss
    03 .tdata .tbss
    04

    readelf -h (UPXed binary):

    ELF Header:
    Magic: 7f 45 4c 46 01 01 01 03 00 00 00 00 00 00 00 00
    Class: ELF32
    Data: 2's complement, little endian
    Version: 1 (current)
    OS/ABI: UNIX - Linux
    ABI Version: 0
    Type: EXEC (Executable file)
    Machine: ARM
    Version: 0x1
    Entry point address: 0x9fe68
    Start of program headers: 52 (bytes into file)
    Start of section headers: 0 (bytes into file)
    Flags: 0x4000002, has entry point, Version4 EABI
    Size of this header: 52 (bytes)
    Size of program headers: 32 (bytes)
    Number of program headers: 2
    Size of section headers: 40 (bytes)
    Number of section headers: 0
    Section header string table index: 0 <corrupt: out of range>

    Section header string table index looks suspicious but I don't know whether that's normal for UPXed bins.

    Uncompressed it looks like this:

    ELF Header:
    Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
    Class: ELF32
    Data: 2's complement, little endian
    Version: 1 (current)
    OS/ABI: UNIX - System V
    ABI Version: 0
    Type: EXEC (Executable file)
    Machine: ARM
    Version: 0x1
    Entry point address: 0x89e0
    Start of program headers: 52 (bytes into file)
    Start of section headers: 1115572 (bytes into file)
    Flags: 0x4000002, has entry point, Version4 EABI
    Size of this header: 52 (bytes)
    Size of program headers: 32 (bytes)
    Number of program headers: 5
    Size of section headers: 40 (bytes)
    Number of section headers: 19
    Section header string table index: 18

     
    Last edit: Dennis 2013-06-05
  • John Reiser

    John Reiser - 2013-06-06

    Which operating system and version is running on the tablet where the UPX'ed busybox gets the segfault?

    Please attach the UPX'ed busybox program which segfaults. Near the bottom of this Tickets page there is a 'Post' button and next to that is 'Cancel' and 'Add attachment'. Click on 'Add attachment', browse and select the UPX'ed busybox file, then Upload. Thank you.

     
  • Dennis

    Dennis - 2013-06-06

    Hi John,

    The tarball contains the upx binaries, busybox, nano and cpuinfo.txt. The tablet runs on android 4.0.3 and debian 7.0 (armv7l). Segfaults on both, also with pre-compiled upx obtained with apt-get.

     
  • Dennis

    Dennis - 2013-06-06

    In addition, gdb output you've requested (on upx'ed nano). Hope this helps:

    (gdb) run
    Starting program: /root/nano

    Program received signal SIGSEGV, Segmentation fault.
    0x0002f000 in ?? ()
    (gdb) info reg
    r0 0x39000 233472
    r1 0x12c34 76852
    r2 0xbefffe9c 3204447900
    r3 0x1000 4096
    r4 0x22 34
    r5 0x4ad0c 306444
    r6 0x4ae84 306820
    r7 0x5b 91
    r8 0xbefff3ac 3204445100
    r9 0xa7 167
    r10 0x3808c 229516
    r11 0x0 0
    r12 0x4ad0c 306444
    sp 0xbefffdc0 0xbefffdc0
    lr 0x400007c0 1073743808
    pc 0x2f000 0x2f000
    cpsr 0x60000010 1610612752
    (gdb) x/8i $pc-4*4
    0x2eff0: andeq r0, r0, r0
    0x2eff4: andeq r0, r0, r0
    0x2eff8: andeq r0, r0, r0
    0x2effc: andeq r0, r0, r0
    => 0x2f000: Cannot access memory at address 0x2f000
    (gdb) info proc
    process 7638
    cmdline = '/root/nano'
    cwd = '/root'
    warning: unable to read link '/proc/7638/exe'
    (gdb) shell cat /proc/7638/maps
    00008000-0002f000 r-xp 00000000 00:00 0
    00036000-00037000 rwxp 00000000 00:00 0 [heap]
    00038000-0004c000 rwxp 00000000 00:00 0
    40000000-4001d000 r-xp 00000000 07:01 41536 /lib/arm-linux-gnueabi/ld-2.13.so
    4001d000-40024000 ---p 00000000 00:00 0
    40024000-40026000 rwxp 0001c000 07:01 41536 /lib/arm-linux-gnueabi/ld-2.13.so
    befdf000-bf000000 rwxp 00000000 00:00 0 [stack]
    ffff0000-ffff1000 r-xp 00000000 00:00 0 [vectors]

     
    Last edit: Dennis 2013-06-06
  • John Reiser

    John Reiser - 2013-06-06

    Thank you, Dennis, for that gdb output! The maps show that the PT_INTERP ld-2.13.so is in the right place, and there are no other shared libraries loaded, not even the libncursesw.so.5, libtinfo.so.5, and libc.so.6 which appear as [DT_]NEEDED in "readelf --all nano-uncompressed". The pc value of 0x2f000 has run off the end of the program, which "readelf --headers nano-uncompressed" shows runs from 0x8000 to 0x2e354. All of this suggests that the problem lies in the transfer of control between the UPX runtime stub (after it has uncompressed the program into the correct addresses) and the PT_INTERP ld-2.13.so which is the "first instruction" of a normal process.

    I will attach a special nano-compressed (about 80KB) which prints some debugging information which I hope will help diagnose what is going wrong. If you can, then please run it under strace with these parameters, and report the output:
    $ strace -e trace=file,munmap,brk,uname,mmap2,mprotect,cacheflush ./nano-upx-debug
    If you don't have strace then please try to install it. If that fails, then just run the program itself:
    $ ./nano-upx-debug

    The output which I see when running under strace looks like this, which indicates a successful transfer of control to ld-linux:

    $ strace -e trace=file,munmap,brk,uname,mmap2,mprotect,cacheflush ./nano-upx-debug
    execve("./nano-upx-debug", ["./nano-upx-debug"], [/ 16 vars /]) = 0

    00000001>
    00000000 00000000 bee76987 00000000 00000000 00000000 00000000 00000000 00000000
    00000008 00000000 00000000 00000000 00000000 00000000 00000000 0001ac54 00000001
    00000010 bee76987 00000000 bee76998 bee769a3 bee769b3 bee769d5 bee769e8 bee769f5
    mmap2(0x38000, 80948, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x38000

    00000002>
    00000000 00038000 00013c34 00000007 00000032 ffffffff 00000000 00000000 000000c0
    00000008 00000000 00008000 0001aed0 0001afac 0001ac34 00000022 0001aca0 00038000
    00000010 00013c34 00038000 00000000 00000022 00000001 bee76987 00000000 bee76998
    cacheflush(0x4ade4, 0x4aee0, 0, 0x2afffffb, 0x30000) = 0

    00000003>
    00000000 0001afb8 00000908 0004aee4 bee76884 00030000 0004ade4 e28f1014 000f0002
    00000008 e3a07004 00012ba8 0003808c 0001afac 0001ac34 0004aee4 0001ad24 00005003
    00000010 0004aee4 00038000 00013c34 00000d58 00000000 00000022 00000001 bee76987
    cacheflush(0x4aee4, 0x4bc3c, 0, 0x4aee4, 0) = 0
    upx_main av=0xbee768e0 szc=0x12ba8 f_dec=0x4ade4 f_unf=0x4af88 xo=0xbee75e60(0x134 0xbee75e7c) xi=0xbee75e68(0xa7 0x3808c) dynbase=0x0
    unpackExtent in=0xbee75e68(0xa7 0x3808c) out=0xbee75e60(0x134 0xbee75e7c) 0x4ade4 0x0
    xread 0xbee75e68(0xa7 0x3808c) 0xbee75df0 0xc
    cacheflush(0xbee75e7c, 0xbee75fb0, 0, 0xbee75e7c, 0) = 0
    auxv_up 0xbee768e0 0x5 0x8
    auxv_up 0xbee768e0 0x4 0x20
    auxv_up 0xbee768e0 0x3 0x8034
    xfind_pages 0x10 0xbee75eb0 8 0xbee75dec
    do_xmap fdi=0x4ade4 ehdr=0xbee75e7c xi=0xbee75e68(0x12ba8 0x3808c) av=0xbee768e0 p_reloc=0xbee75e38 f_unf=0x4af88
    auxv_up 0xbee768e0 0x3 0x8034
    mmap2(0x8000, 156500, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x8000
    unpackExtent in=0xbee75e68(0x12ba8 0x3808c) out=0xbee75de4(0x26354 0x8000) 0x4ade4 0x4af88
    xread 0xbee75e68(0x12ba8 0x3808c) 0xbee75d90 0xc
    cacheflush(0x8000, 0x8134, 0, 0x8000, 0) = 0
    xread 0xbee75e68(0x12b01 0x38133) 0xbee75d90 0xc
    cacheflush(0x8134, 0x2e354, 0, 0x8134, 0) = 0
    make_hatch 0xbee75f10 0x0
    auxv_up 0xbee768e0 0x0 0x2e354
    mprotect(0x8000, 156500, PROT_READ|PROT_EXEC) = 0
    mmap2(0x36000, 1896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x36000
    unpackExtent in=0xbee75e68(0xf8 0x4ab3c) out=0xbee75de4(0x414 0x36354) 0x4ade4 0x0
    xread 0xbee75e68(0xf8 0x4ab3c) 0xbee75d90 0xc
    cacheflush(0x36354, 0x36768, 0, 0x36354, 0) = 0
    make_hatch 0xbee75f30 0x0
    mprotect(0x36000, 1896, PROT_READ|PROT_WRITE) = 0
    brk(0x37000) = 0x37000
    auxv_up 0xbee768e0 0x9 0xb694
    open("/lib/ld-linux.so.3", O_RDONLY) = 3
    xfind_pages 0x0 0xbee75eb0 8 0xbee75dec
    mmap2(NULL, 155648, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40000000
    do_xmap fdi=0x3 ehdr=0xbee75e7c xi=0x0(0x0 0x0) av=0xbee768e0 p_reloc=0xbee75e38 f_unf=0x0
    mmap2(0x40000000, 115920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0x40000000
    mmap2(0x40024000, 6236, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1c) = 0x40024000
    auxv_up 0xbee768e0 0x7 0x40000000
    munmap(0x38000, 80948) = 0

    here<< is the transfer of control from upx stub to PT_INTERP

    brk(0) = 0x37000
    uname({sys="Linux", node="sheevaplug", ...}) = 0
    access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
    mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40026000
    access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
    open("/etc/ld.so.cache", O_RDONLY) = 3
    fstat64(3, {st_mode=S_IFREG|0644, st_size=50687, ...}) = 0
    mmap2(NULL, 50687, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40027000
    access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
    open("/lib/libncursesw.so.5", O_RDONLY) = 3
    fstat64(3, {st_mode=S_IFREG|0644, st_size=272220, ...}) = 0
    mmap2(NULL, 271820, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40034000
    mmap2(0x40074000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x40) = 0x40074000
    [[snip]]


     
  • Dennis

    Dennis - 2013-06-06

    Hi John,

    On segfault:

    strace:

    root@evil:~# strace -e trace=file,munmap,brk,uname,mmap2,mprotect,cacheflush ./nano-upx-debug
    execve("./nano-upx-debug", ["./nano-upx-debug"], [/ 13 vars /]) = 0

    00000001>
    00000000 00000000 bea37eb7 00000000 00000000 00000000 00000000 00000000 00000000
    00000008 00000000 00000000 00000000 00000000 00000000 00000000 0001ac54 00000001
    00000010 bea37eb7 00000000 bea37ec8 bea37ed3 bea37ee3 bea37f06 bea37f19 bea37f23
    mmap2(0x38000, 80948, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x38000

    00000002>
    00000000 00038000 00013c34 00000007 00000032 ffffffff 00000000 00000000 000000c0
    00000008 00000000 00008000 0001aed0 0001afac 0001ac34 00000022 0001aca0 00038000
    00000010 00013c34 00038000 00000000 00000022 00000001 bea37eb7 00000000 bea37ec8
    cacheflush(0x4ade4, 0x4aee0, 0, 0x2afffffb, 0x30000) = 0

    00000003>
    00000000 0001afb8 00000908 0004aee4 bea37db4 00030000 0004ade4 e28f1014 000f0002
    00000008 e3a07004 00012ba8 0003808c 0001afac 0001ac34 0004aee4 0001ad24 00005003
    00000010 0004aee4 00038000 00013c34 00000d58 00000000 00000022 00000001 bea37eb7
    cacheflush(0x4aee4, 0x4bc3c, 0, 0x4aee4, 0) = 0
    upx_main av=0xbea37e04 szc=0x12ba8 f_dec=0x4ade4 f_unf=0x4af88 xo=0xbea37390(0x134 0xbea373ac) xi=0xbea37398(0xa7 0x3808c) dynbase=0x0
    unpackExtent in=0xbea37398(0xa7 0x3808c) out=0xbea37390(0x134 0xbea373ac) 0x4ade4 0x0
    xread 0xbea37398(0xa7 0x3808c) 0xbea37320 0xc
    cacheflush(0xbea373ac, 0xbea374e0, 0, 0xbea373ac, 0) = 0
    auxv_up 0xbea37e04 0x5 0x8
    auxv_up 0xbea37e04 0x4 0x20
    auxv_up 0xbea37e04 0x3 0x8034
    xfind_pages 0x10 0xbea373e0 8 0xbea3731c
    do_xmap fdi=0x4ade4 ehdr=0xbea373ac xi=0xbea37398(0x12ba8 0x3808c) av=0xbea37e04 p_reloc=0xbea37368 f_unf=0x4af88
    auxv_up 0xbea37e04 0x3 0x8034
    mmap2(0x8000, 156500, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x8000
    unpackExtent in=0xbea37398(0x12ba8 0x3808c) out=0xbea37314(0x26354 0x8000) 0x4ade4 0x4af88
    xread 0xbea37398(0x12ba8 0x3808c) 0xbea372c0 0xc
    cacheflush(0x8000, 0x8134, 0, 0x8000, 0) = 0
    xread 0xbea37398(0x12b01 0x38133) 0xbea372c0 0xc
    cacheflush(0x8134, 0x2e354, 0, 0x8134, 0) = 0
    make_hatch 0xbea37440 0x0
    auxv_up 0xbea37e04 0x0 0x2e354
    mprotect(0x8000, 156500, PROT_READ|PROT_EXEC) = 0
    mmap2(0x36000, 1896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x36000
    unpackExtent in=0xbea37398(0xf8 0x4ab3c) out=0xbea37314(0x414 0x36354) 0x4ade4 0x0
    xread 0xbea37398(0xf8 0x4ab3c) 0xbea372c0 0xc
    cacheflush(0x36354, 0x36768, 0, 0x36354, 0) = 0
    make_hatch 0xbea37460 0x0
    mprotect(0x36000, 1896, PROT_READ|PROT_WRITE) = 0
    brk(0x37000) = 0xd96000
    auxv_up 0xbea37e04 0x9 0xb694
    open("/lib/ld-linux.so.3", O_RDONLY) = 3
    xfind_pages 0x0 0xbea373e0 7 0xbea3731c
    mmap2(NULL, 155648, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4006f000
    do_xmap fdi=0x3 ehdr=0xbea373ac xi=0x0(0x0 0x0) av=0xbea37e04 p_reloc=0xbea37368 f_unf=0x0
    mmap2(0x4006f000, 118032, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0x4006f000
    mmap2(0x40093000, 6260, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1c) = 0x40093000
    auxv_up 0xbea37e04 0x7 0x4006f000
    --- SIGSEGV (Segmentation fault) @ 0 (0) ---
    +++ killed by SIGSEGV +++
    Segmentation fault

    Debug output:

    root@evil:~# ./nano-upx-debug --help

    00000001>
    00000000 00000000 be9c3eaf be9c3ec0 00000000 00000000 00000000 00000000 00000000
    00000008 00000000 00000000 00000000 00000000 00000000 00000000 0001ac54 00000002
    00000010 be9c3eaf be9c3ec0 00000000 be9c3ec7 be9c3ed2 be9c3ee2 be9c3f05 be9c3f18

    00000002>
    00000000 00038000 00013c34 00000007 00000032 ffffffff 00000000 00000000 000000c0
    00000008 00000000 00008000 0001aed0 0001afac 0001ac34 00000022 0001aca0 00038000
    00000010 00013c34 00038000 00000000 00000022 00000002 be9c3eaf be9c3ec0 00000000

    00000003>
    00000000 0001afb8 00000908 0004aee4 be9c3db4 00030000 0004ade4 e28f1014 000f0002
    00000008 e3a07004 00012ba8 0003808c 0001afac 0001ac34 0004aee4 0001ad24 00005003
    00000010 0004aee4 00038000 00013c34 00000d58 00000000 00000022 00000002 be9c3eaf
    upx_main av=0xbe9c3e08 szc=0x12ba8 f_dec=0x4ade4 f_unf=0x4af88 xo=0xbe9c3390(0x134 0xbe9c33ac) xi=0xbe9c3398(0xa7 0x3808c) dynbase=0x0
    unpackExtent in=0xbe9c3398(0xa7 0x3808c) out=0xbe9c3390(0x134 0xbe9c33ac) 0x4ade4 0x0
    xread 0xbe9c3398(0xa7 0x3808c) 0xbe9c3320 0xc
    auxv_up 0xbe9c3e08 0x5 0x8
    auxv_up 0xbe9c3e08 0x4 0x20
    auxv_up 0xbe9c3e08 0x3 0x8034
    xfind_pages 0x10 0xbe9c33e0 8 0xbe9c331c
    do_xmap fdi=0x4ade4 ehdr=0xbe9c33ac xi=0xbe9c3398(0x12ba8 0x3808c) av=0xbe9c3e08 p_reloc=0xbe9c3368 f_unf=0x4af88
    auxv_up 0xbe9c3e08 0x3 0x8034
    unpackExtent in=0xbe9c3398(0x12ba8 0x3808c) out=0xbe9c3314(0x26354 0x8000) 0x4ade4 0x4af88
    xread 0xbe9c3398(0x12ba8 0x3808c) 0xbe9c32c0 0xc
    xread 0xbe9c3398(0x12b01 0x38133) 0xbe9c32c0 0xc
    make_hatch 0xbe9c3440 0x0
    auxv_up 0xbe9c3e08 0x0 0x2e354
    unpackExtent in=0xbe9c3398(0xf8 0x4ab3c) out=0xbe9c3314(0x414 0x36354) 0x4ade4 0x0
    xread 0xbe9c3398(0xf8 0x4ab3c) 0xbe9c32c0 0xc
    make_hatch 0xbe9c3460 0x0
    auxv_up 0xbe9c3e08 0x9 0xb694
    xfind_pages 0x0 0xbe9c33e0 7 0xbe9c331c
    do_xmap fdi=0x3 ehdr=0xbe9c33ac xi=0x0(0x0 0x0) av=0xbe9c3e08 p_reloc=0xbe9c3368 f_unf=0x0
    auxv_up 0xbe9c3e08 0x7 0x400d5000
    Segmentation fault

     
  • John Reiser

    John Reiser - 2013-06-07

    Hi Dennis, we are making progress.

    The line from the strace:
    brk(0x37000) = 0xd96000
    is worrisome. The output result 0xd96000 should be the same as the input argument 0x37000. But perhaps this is some Address Space Layout Randomization that is not supposed to matter. I'll have to think about it.

    Meanwhile, attached is another compressed nano to run under strace:
    $ strace -e trace=file,munmap,brk,uname,mmap2,mprotect,cacheflush ./nano-upx-debug2
    The output should be much the same as before, except for an added section at the end which shows where the handoff will be executed (0002e354) and the instructions that will do it (ef000000 e1a0f00e). The munmap is the last operation in the UPX stub, and the brk(0) is the first operation in ld-linux.


    auxv_up 0xbec228e0 0x7 0x40000000
    cacheflush(0x2e354, 0x2e35c, 0, 0x7, 0x12ba8) = 0

    00000004>
    00000000 00038000 00013c34 bec22970 0002e354 ef000000 e1a0f00e 0004afbc 000f0002
    00000008 bec21e7c 000000a7 0003808c 00000000 00012ba8 400007e0 0004af90 00000001
    00000010 bec22985 00000000 bec22997 bec229a2 bec229b2 bec229d4 bec229e7 bec229f4
    munmap(0x38000, 80948) = 0
    brk(0) = 0x37000
    uname({sys="Linux", node="sheevaplug", ...}) = 0


    (The first nano-upx-debug is no longer needed and may be discarded.)

     
  • Dennis

    Dennis - 2013-06-07

    Hi John,

    Your new upx'ed nano refuses to segfault. It seems you solved it on your end. Do you have a diff patch ?

    strace output:

    00000001>
    00000000 00000000 beb90eb3 00000000 00000000 00000000 00000000 00000000 00000000
    00000008 00000000 00000000 00000000 00000000 00000000 00000000 0001ac54 00000001
    00000010 beb90eb3 00000000 beb90ec5 beb90ed0 beb90ee0 beb90f03 beb90f16 beb90f20

    00000002>
    00000000 00038000 00013c34 00000007 00000032 ffffffff 00000000 00000000 000000c0
    00000008 00000000 00008000 0001aed0 0001afac 0001ac34 00000022 0001aca0 00038000
    00000010 00013c34 00038000 00000000 00000022 00000001 beb90eb3 00000000 beb90ec5

    00000003>
    00000000 0001afb8 000009af 0004aee4 beb90db4 00030000 0004ade4 e28f1014 000f0002
    00000008 e3a07004 00012ba8 0003808c 0001afac 0001ac34 0004aee4 0001ad24 00005003
    00000010 0004aee4 00038000 00013c34 00000e48 00000000 00000022 00000001 beb90eb3
    upx_main av=0xbeb90e04 szc=0x12ba8 f_dec=0x4ade4 f_unf=0x4afbc xo=0xbeb90390(0x134 0xbeb903ac) xi=0xbeb90398(0xa7 0x3808c) dynbase=0x0
    unpackExtent in=0xbeb90398(0xa7 0x3808c) out=0xbeb90390(0x134 0xbeb903ac) 0x4ade4 0x0
    xread 0xbeb90398(0xa7 0x3808c) 0xbeb90320 0xc
    auxv_up 0xbeb90e04 0x5 0x8
    auxv_up 0xbeb90e04 0x4 0x20
    auxv_up 0xbeb90e04 0x3 0x8034
    xfind_pages 0x10 0xbeb903e0 8 0xbeb9031c
    do_xmap fdi=0x4ade4 ehdr=0xbeb903ac xi=0xbeb90398(0x12ba8 0x3808c) av=0xbeb90e04 p_reloc=0xbeb90368 f_unf=0x4afbc
    auxv_up 0xbeb90e04 0x3 0x8034
    unpackExtent in=0xbeb90398(0x12ba8 0x3808c) out=0xbeb90314(0x26354 0x8000) 0x4ade4 0x4afbc
    xread 0xbeb90398(0x12ba8 0x3808c) 0xbeb902c0 0xc
    xread 0xbeb90398(0x12b01 0x38133) 0xbeb902c0 0xc
    make_hatch 0xbeb90440 0x0 0xef000000
    auxv_up 0xbeb90e04 0x0 0x2e354
    unpackExtent in=0xbeb90398(0xf8 0x4ab3c) out=0xbeb90314(0x414 0x36354) 0x4ade4 0x0
    xread 0xbeb90398(0xf8 0x4ab3c) 0xbeb902c0 0xc
    make_hatch 0xbeb90460 0x0 0xef000000
    auxv_up 0xbeb90e04 0x9 0xb694
    xfind_pages 0x0 0xbeb903e0 7 0xbeb9031c
    do_xmap fdi=0x3 ehdr=0xbeb903ac xi=0x0(0x0 0x0) av=0xbeb90e04 p_reloc=0xbeb90368 f_unf=0x0
    auxv_up 0xbeb90e04 0x7 0x400e3000

     
  • John Reiser

    John Reiser - 2013-06-07

    Hi Dennis,
    Mercurial ("hg") revision 0c159f733586 "ARM __clear_cache() on escape hatch" is the fix. I also attach the "hg diff" here.

    Thank you for your help in debugging. I appreciate it!

     
  • John Reiser

    John Reiser - 2013-06-07
    • status: open --> pending
    • assigned_to: John Reiser
    • Group: -->
     
  • Dennis

    Dennis - 2013-06-07

    Hi John,

    I can confirm the patch worked for my debian compilation of UPX 3.10, but not for every file. Nano seems to work just fine. I've then tested with busybox, where it occasionally segfaults:

    execve("./tinybox", ["./tinybox"], [/ 33 vars /]) = 0
    mmap2(0x208000, 1024252, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x208000
    cacheflush(0x3011d4, 0x301320, 0, 0x1afffffb, 0x200000) = 0
    cacheflush(0x301324, 0x301bcc, 0, 0x301324, 0) = 0
    cacheflush(0xbed45fbc, 0xbed460b0, 0, 0xbed45fbc, 0) = 0
    mmap2(0x8000, 1882976, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x8000
    cacheflush(0x8000, 0x80f4, 0, 0x8000, 0) = 0
    cacheflush(0x80f4, 0x1d3b60, 0, 0x80f4, 0) = 0
    mprotect(0x8000, 1882976, PROT_READ|PROT_EXEC) = 0
    mmap2(0x1db000, 5411, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x1db000
    cacheflush(0x1dbb60, 0x1dc523, 0, 0x1dbb60, 0) = 0
    mprotect(0x1db000, 5411, PROT_READ|PROT_WRITE) = 0
    mmap2(0x1dd000, 174976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x1dd000
    brk(0x208000) = 0x148c000
    cacheflush(0x1d3b60, 0x1d3b68, 0, 0x6474e551, 0xf9070) = 0
    munmap(0x208000, 1024252) = 0
    uname({sys="Linux", node="evil", ...}) = 0
    brk(0) = 0x148c000
    brk(0x148cd24) = 0x148cd24
    brk(0x14add24) = 0x14add24
    brk(0x14ae000) = 0x14ae000
    --- SIGSEGV (Segmentation fault) @ 0 (0) ---
    +++ killed by SIGSEGV +++

     
  • John Reiser

    John Reiser - 2013-06-07

    Hi Dennis, The SIGSEGV on busybox is disappointing.

    Would you please run under gdb and print the reginfo, the 8 instructions around the pc, and the /proc/PID/maps , like before with nano?

     
  • Dennis

    Dennis - 2013-06-07

    Hi John,

    Note that the binary I posted the previous strace of differs from the busybox binary I've sent you earlier. I've now ran an strace on the binary I've attached previously (ofcourse I've compressed it with the patched UPX).

    If it'd be easier to debug for you I can set you up with SSH/root access. Just tell me how to contact you directly.

    Results are as follows:

    gdb:

    (gdb) run
    Starting program: /root/busybox.reiser

    Program received signal SIGSEGV, Segmentation fault.
    0x00008fac in ?? ()
    (gdb) info reg
    r0 0x275e4 161252
    r1 0xbefffda4 3204447652
    r2 0x4ebc8 322504
    r3 0xf6da8 1011112
    r4 0xbefffea3 3204447907
    r5 0xbefffda4 3204447652
    r6 0x80d4 32980
    r7 0x118004 1146884
    r8 0x0 0
    r9 0x1 1
    r10 0x0 0
    r11 0xbefffda4 3204447652
    r12 0x2eb0 11952
    sp 0xbefffba8 0xbefffba8
    lr 0x91f0 37360
    pc 0x8fac 0x8fac
    cpsr 0x80000010 -2147483632
    (gdb) x/8i $pc-4*4
    0x8f9c: ldr r3, [sp, #4]
    0x8fa0: cmp r9, #2
    0x8fa4: lsl r2, r3, #1
    0x8fa8: ldr r3, [pc, #484] ; 0x9194
    => 0x8fac: ldrh r5, [r3, r2]
    0x8fb0: ldr r2, [pc, #480] ; 0x9198
    0x8fb4: lsl r3, r5, #20
    0x8fb8: lsr r3, r3, #20
    (gdb) info proc
    process 19053
    cmdline = '/root/busybox.reiser'
    cwd = '/root'
    warning: unable to read link '/proc/19053/exe'
    (gdb) shell cat /proc/19053/maps
    00008000-00118000 r-xp 00000000 00:00 0
    00118000-00120000 rwxp 00000000 00:00 0 [heap]
    befdf000-bf000000 rwxp 00000000 00:00 0 [stack]
    ffff0000-ffff1000 r-xp 00000000 00:00 0 [vectors]

    strace:

    execve("./busybox.reiser", ["./busybox.reiser"], [/ 14 vars /]) = 0
    mmap2(0x120000, 626260, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x120000
    cacheflush(0x1b7f2c, 0x1b8080, 0, 0xe3a0201e, 0x118000) = 0
    cacheflush(0x1b8084, 0x1b892c, 0, 0x1b8084, 0) = 0
    cacheflush(0xbe8eb39c, 0xbe8eb470, 0, 0xbe8eb39c, 0) = 0
    mmap2(0x8000, 1113484, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x8000
    cacheflush(0x8000, 0x80d4, 0, 0x8000, 0) = 0
    cacheflush(0x80d4, 0x117d8c, 0, 0x80d4, 0) = 0
    mprotect(0x8000, 1113484, PROT_READ|PROT_EXEC) = 0
    mmap2(0x118000, 1254, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x118000
    cacheflush(0x118000, 0x1184e6, 0, 0x118000, 0) = 0
    mprotect(0x118000, 1254, PROT_READ|PROT_WRITE) = 0
    mmap2(0x119000, 17248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x119000
    brk(0x11e000) = 0x1a8c000
    cacheflush(0x117d8c, 0x117d94, 0, 0x6474e551, 0x97dc8) = 0
    munmap(0x120000, 626260) = 0
    brk(0) = 0x1a8c000
    brk(0x1a8c4b8) = 0x1a8c4b8
    mmap2(NULL, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0) = 0x400c1000
    --- SIGSEGV (Segmentation fault) @ 0 (0) ---
    +++ killed by SIGSEGV +++
    Segmentation fault

     
  • John Reiser

    John Reiser - 2013-06-07

    Hi Dennis,

    Yes, I would like to try ssh remote access. If you click on my name (under any instance of the 1/0 icon) there's a page with two addresses for me. I've used ssh to someone else's machine before, but not recently, so in your private mail to me then please remind me of how to set up the key. Thank you.

     
  • Dennis

    Dennis - 2013-06-07
     
    Last edit: Dennis 2013-06-08
  • John Reiser

    John Reiser - 2013-06-13

    Summary: cacheflush() was not being applied to the "escape hatch". That was fixed. Even then, statically-linked programs (not using ld-linux.so) still suffered SIGSEGV. So there is a bug in that Linux kernel version 3.0.6, or in a mismatch between .config and CPU, or etc. As a workaround, I installed a heuristic "sweep" of 1/2 MB of stack space, deliberately causing collisions and evictions from the cache. (The execution of ld-linux.so as PT_INTERP for a dyanmically-linked program was enough to do this flush as an inadvertent side effect.) This works for now, but may fail for new CPUs in the future. By then perhaps the bugs will have died out.

    The decompressor stub also was enhanced to remember the link text of /proc/self/exe in a new environment variable whose name is " " [three spaces]. Because nothing can be mapped below 0x8000 (32KB), and keeping brk(0) sane requires nothing above the original end of .data+.bss, then /proc/self/exe becomes "(deleted)" as the decompressor unmaps itself. Too bad: the existence of /proc/self/exe is not guaranteed, and there are other reasons why it may vanish, too.

     
  • John Reiser

    John Reiser - 2013-06-13
    • status: pending --> closed-fixed
     

Log in to post a comment.