#61 mpd5 crashes (Illegal instruction) when running from a VM

[Olivier Cochard-Labbe]

mpd5 crashes (Illegal instruction) when running from a VM:
Tested under bhyve (host using fbsd -head r331837) and VirtualBox 5.1(host using Windows 7).
Here is a simple configuration file for easy reproducing of the crash (but anykind of config will crash it)

[root@bhyvevm]~# ifconfig em0 1.1.1.1/24 up
[root@bhyvevm]~# cat > /usr/local/etc/mpd5/mpd.conf <<'EOF'
default:
        create bundle static B1
        set bundle enable ipv6cp
        set ipcp enable req-pri-dns  
        set ipcp enable req-sec-dns  
        set iface route default
        create link static L1 pppoe
        set link action bundle B1
        set auth authname olivier
        set auth password secret
        set pppoe iface em0
        open
'EOF'

[root@bhyvevm]~# mpd5
Multi-link PPP daemon for FreeBSD

process 55152 started, version 5.8 (root@lame4.bsdrp.net 15:49  1-Apr-2018)
WARNING: attempt to domain_add(netgraph) after domainfinalize()
Illegal instruction

The same raw image disk (same FreeBSD version and mpd5) once transferet on a real hardware didn't crash anymore:

[root@apu2]~# ifconfig igb2 1.1.1.1/24 up
[root@apu2]~# cat > /usr/local/etc/mpd5/mpd.conf <<'EOF'
default:
        create bundle static B1
        set bundle enable ipv6cp
        set ipcp enable req-pri-dns  
        set ipcp enable req-sec-dns  
        set iface route default
        create link static L1 pppoe
        set link action bundle B1
        set auth authname olivier
        set auth password secret
        set pppoe iface igb2
        open
'EOF'

[root@apu2]~# mpd5
Multi-link PPP daemon for FreeBSD

process 908 started, version 5.8 (root@lame4.bsdrp.net 15:49  1-Apr-2018)
Label 'startup' not found
[B1] Bundle: Interface ng0 created
[L1] [L1] Link: OPEN event
[L1] LCP: Open event
[L1] LCP: state change Initial --> Starting
[L1] LCP: LayerStart
[L1] PPPoE: Connecting to '*'

[Eugene Grosbein]

"Illegal instruction" probably means you use mpd package built with CPU optimizations for higher class CPU that is emulated by hypervisor. Try rebuilding mpd5 without any extra optimizations at all, e.g. default only; empty src.conf and make.conf

[Olivier Cochard-Labbe]

Thanks for the clue!
I didn't have any extra optimization into src.conf neither make.conf.
But for confirming your idea I've downloaded official -current (FreeBSD-12.0-CURRENT-amd64-20180329-r331740) and installed the mpd5 packages from official pkg repo: No more "illegal instruction" on my VM.
I need to dig more the way I'm building this package on my side, but there is something strange somewhere.

[Olivier Cochard-Labbe]

I've found the problem: It's not an extra optimization.
It's a "security.bsd.stack_guard_page=512" on my etc/sysctl.conf.
Try it on a bhyve/virtualbox VM, and mpd5 will crash.