kernel punic ! please please help :( !

Help
changizy
2012-09-02
2013-03-27
  • changizy
    changizy
    2012-09-02

    hello ! it is about 2 month i am trying to create server with more than 7000 pppoe user ! every thing is working ! but i get kernel punics :( i tryed many hardware ! there is no SHAPING or NAT ! just FreeBSD 9.0, mpd, and routing for internet access !! when i use polling it punics faster ! about 10 min uptime ! and when i remove polling about one 1 Hour and punic again :(

    hardware i used :
    1.ASUS MOTHERBOARD + 8GB DDR3 RAM + INTEL I5 CPU 3.3 ,  2 intel (em) Interfaces
    2. IBM SERVER , with 2 Cpu 2.0 Mhz , 8 GB DDR2 RAM , 2 Brodcam (bce) Interfaces

    cpu usage when crashed in one core is about 40% idle (all  used by system) and all other cores is about 70% or 80% idle
    here one! of the punics in IBM SERVER!

    PPPOESERVER dumped core - see /var/crash/vmcore.0
    Mon Aug 27 03:08:40 IRDT 2012
    FreeBSD PPPOESERVER 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Aug 21 09:15:30 IRDT 2012     root@pppoe:/usr/obj/usr/src/sys/GENERIC  amd64
    panic: page fault
    GNU gdb 6.1.1 [FreeBSD]
    Copyright 2004 Free Software Foundation, Inc.
    GDB is free software, covered by the GNU General Public License, and you are
    welcome to change it and/or distribute copies of it under certain conditions.
    Type "show copying" to see the conditions.
    There is absolutely no warranty for GDB.  Type "show warranty" for details.
    This GDB was configured as "amd64-marcel-freebsd"...
    Unread portion of the kernel message buffer:
    Fatal trap 12: page fault while in kernel mode
    cpuid = 0; apic id = 00
    fault virtual address   = 0x10000fa5c
    fault code      = supervisor read data, page not present
    instruction pointer = 0x20:0xffffffff809ea451
    stack pointer           = 0x28:0xffffff81f0f56aa0
    frame pointer           = 0x28:0xffffff81f0f56ad0
    code segment        = base 0x0, limit 0xfffff, type 0x1b
                = DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags    = interrupt enabled, resume, IOPL = 0
    current process     = 0 (em0 taskq)
    trap number     = 12
    panic: page fault
    cpuid = 0
    KDB: stack backtrace:
    #0 0xffffffff8086b1de at kdb_backtrace+0x5e
    #1 0xffffffff80835d97 at panic+0x187
    #2 0xffffffff80b40ec0 at trap_fatal+0x290
    #3 0xffffffff80b41209 at trap_pfault+0x1f9
    #4 0xffffffff80b416cf at trap+0x3df
    #5 0xffffffff80b2bbff at calltrap+0x8
    #6 0xffffffff809ea661 at HouseKeeping+0xa1
    #7 0xffffffff809e78fb at LibAliasOutLocked+0x3b
    #8 0xffffffff809e97fa at LibAliasOut+0x5a
    #9 0xffffffff80963fb9 at ipfw_nat+0x99
    #10 0xffffffff8095abf7 at ipfw_chk+0x7a7
    #11 0xffffffff809607d6 at ipfw_check_hook+0xf6
    #12 0xffffffff808f5bbe at pfil_run_hooks+0x9e
    #13 0xffffffff80968e4f at ip_output+0x4ff
    #14 0xffffffff80965ec3 at ip_forward+0x303
    #15 0xffffffff80967578 at ip_input+0x5c8
    #16 0xffffffff808f4e8b at netisr_dispatch_src+0x20b
    #17 0xffffffff8161ad94 at ng_iface_rcvdata+0x104
    Uptime: 1h59m49s
    Dumping 669 out of 8154 MB:..3%..12%..22%..32%..41%..51%..63%..72%..82%..91%
    Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_socket.ko
    Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_mppc.ko
    Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/rc4.ko
    Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_ether.ko
    Reading symbols from /boot/kernel/ng_tee.ko...Reading symbols from /boot/kernel/ng_tee.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_tee.ko
    Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from /boot/kernel/ng_iface.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_iface.ko
    Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_ppp.ko
    Reading symbols from /boot/kernel/ng_vjc.ko...Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_vjc.ko
    #0  doadump (textdump=Variable "textdump" is not available.
    ) at pcpu.h:224
    224 pcpu.h: No such file or directory.
        in pcpu.h
    (kgdb) #0  doadump (textdump=Variable "textdump" is not available.
    ) at pcpu.h:224
    #1  0xffffffff808358d5 in kern_reboot (howto=260)
        at /usr/src/sys/kern/kern_shutdown.c:442
    #2  0xffffffff80835d81 in panic (fmt=Variable "fmt" is not available.
    )
        at /usr/src/sys/kern/kern_shutdown.c:607
    #3  0xffffffff80b40ec0 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
    )
        at /usr/src/sys/amd64/amd64/trap.c:819
    #4  0xffffffff80b41209 in trap_pfault (frame=0xffffff81f0f569f0, usermode=0)
        at /usr/src/sys/amd64/amd64/trap.c:735
    #5  0xffffffff80b416cf in trap (frame=0xffffff81f0f569f0)
        at /usr/src/sys/amd64/amd64/trap.c:474
    #6  0xffffffff80b2bbff in calltrap ()
        at /usr/src/sys/amd64/amd64/exception.S:228
    #7  0xffffffff809ea451 in DeleteLink (lnk=0xfffffe00402ca680)
        at /usr/src/sys/netinet/libalias/alias_db.c:859
    #8  0xffffffff809ea661 in HouseKeeping (la=0xffffff8001942000)
        at /usr/src/sys/netinet/libalias/alias_db.c:849
    #9  0xffffffff809e78fb in LibAliasOutLocked (la=0xffffff8001942000, 
        ptr=0xfffffe00373aa016 "E", maxpacketsize=2026, create=1)
        at /usr/src/sys/netinet/libalias/alias.c:1451
    #10 0xffffffff809e97fa in LibAliasOut (la=0xffffff8001942000, 
        ptr=0xfffffe00373aa016 "E", maxpacketsize=2026)
        at /usr/src/sys/netinet/libalias/alias.c:1418
    #11 0xffffffff80963fb9 in ipfw_nat (args=0xffffff81f0f56e90, 
        t=0xfffffe003786d900, m=0xfffffe009bd96e00)
        at /usr/src/sys/netinet/ipfw/ip_fw_nat.c:295
    #12 0xffffffff8095abf7 in ipfw_chk (args=0xffffff81f0f56e90)
        at /usr/src/sys/netinet/ipfw/ip_fw2.c:2358
    #13 0xffffffff809607d6 in ipfw_check_hook (arg=Variable "arg" is not available.
    )
        at /usr/src/sys/netinet/ipfw/ip_fw_pfil.c:141
    #14 0xffffffff808f5bbe in pfil_run_hooks (ph=Variable "ph" is not available.
    ) at /usr/src/sys/net/pfil.c:82
    #15 0xffffffff80968e4f in ip_output (m=0xfffffe009bd96e00, opt=Variable "opt" is not available.
    )
        at /usr/src/sys/netinet/ip_output.c:508
    #16 0xffffffff80965ec3 in ip_forward (m=0xfffffe009bd96e00, srcrt=Variable "srcrt" is not available.
    )
        at /usr/src/sys/netinet/ip_input.c:1494
    #17 0xffffffff80967578 in ip_input (m=0xfffffe009bd96e00)
        at /usr/src/sys/netinet/ip_input.c:702
    #18 0xffffffff808f4e8b in netisr_dispatch_src (proto=1, source=Variable "source" is not available.
    )
        at /usr/src/sys/net/netisr.c:1013
    #19 0xffffffff8161ad94 in ng_iface_rcvdata (hook=Variable "hook" is not available.
    )
        at /usr/src/sys/modules/netgraph/iface/../../../netgraph/ng_iface.c:781
    #20 0xffffffff80938fdb in ng_apply_item (node=0xfffffe003519a600, 
        item=0xfffffe002adf6100, rw=0) at /usr/src/sys/netgraph/ng_base.c:2318
    #21 0xffffffff80937f8e in ng_snd_item (item=Variable "item" is not available.
    )
        at /usr/src/sys/netgraph/ng_base.c:2235
    #22 0xffffffff8161d418 in ng_ppp_rcvdata (hook=Variable "hook" is not available.
    )
        at /usr/src/sys/modules/netgraph/ppp/../../../netgraph/ng_ppp.c:1522
    #23 0xffffffff80938fdb in ng_apply_item (node=0xfffffe00355b5200, 
        item=0xfffffe002adf6100, rw=0) at /usr/src/sys/netgraph/ng_base.c:2318
    #24 0xffffffff80937f8e in ng_snd_item (item=Variable "item" is not available.
    )
        at /usr/src/sys/netgraph/ng_base.c:2235
    #25 0xffffffff80938fdb in ng_apply_item (node=0xfffffe003a854900, 
        item=0xfffffe002adf6100, rw=0) at /usr/src/sys/netgraph/ng_base.c:2318
    #26 0xffffffff80937f8e in ng_snd_item (item=Variable "item" is not available.
    )
        at /usr/src/sys/netgraph/ng_base.c:2235
    #27 0xffffffff8093d8ec in ng_pppoe_rcvdata_ether (hook=Variable "hook" is not available.
    )
        at /usr/src/sys/netgraph/ng_pppoe.c:1655
    #28 0xffffffff80938fdb in ng_apply_item (node=0xfffffe0037bc5100, 
        item=0xfffffe002adf6100, rw=0) at /usr/src/sys/netgraph/ng_base.c:2318
    #29 0xffffffff80937f8e in ng_snd_item (item=Variable "item" is not available.
    )
        at /usr/src/sys/netgraph/ng_base.c:2235
    #30 0xffffffff808ea897 in ether_demux (ifp=0xfffffe00370b9800, m=Variable "m" is not available.
    )
        at /usr/src/sys/net/if_ethersubr.c:954
    #31 0xffffffff808eab94 in ether_nh_input (m=Variable "m" is not available.
    )
        at /usr/src/sys/net/if_ethersubr.c:756
    #32 0xffffffff808f4e8b in netisr_dispatch_src (proto=9, source=Variable "source" is not available.
    )
        at /usr/src/sys/net/netisr.c:1013
    #33 0xffffffff808ea7df in ether_demux (ifp=0xfffffe00064ea800, 
        m=0xfffffe009bd96e00) at /usr/src/sys/net/if_ethersubr.c:846
    #34 0xffffffff808eab94 in ether_nh_input (m=Variable "m" is not available.
    )
        at /usr/src/sys/net/if_ethersubr.c:756
    #35 0xffffffff808f4e8b in netisr_dispatch_src (proto=9, source=Variable "source" is not available.
    )
        at /usr/src/sys/net/netisr.c:1013
    #36 0xffffffff8047303a in em_rxeof (rxr=0xfffffe0006501600, count=99, 
        done=0x0) at /usr/src/sys/dev/e1000/if_em.c:4340
    #37 0xffffffff804734bb in em_handle_que (context=Variable "context" is not available.
    )
        at /usr/src/sys/dev/e1000/if_em.c:1518
    #38 0xffffffff808774e5 in taskqueue_run_locked (queue=0xfffffe00064ee400)
        at /usr/src/sys/kern/subr_taskqueue.c:308
    #39 0xffffffff80878466 in taskqueue_thread_loop (arg=Variable "arg" is not available.
    )
        at /usr/src/sys/kern/subr_taskqueue.c:497
    #40 0xffffffff8080990f in fork_exit (
        callout=0xffffffff80878420 <taskqueue_thread_loop>, 
        arg=0xffffff80012e6748, frame=0xffffff81f0f57c50)
        at /usr/src/sys/kern/kern_fork.c:995
    #41 0xffffffff80b2c12e in fork_trampoline ()
        at /usr/src/sys/amd64/amd64/exception.S:602
    #42 0x0000000000000000 in ?? ()
    #43 0x0000000000000000 in ?? ()
    #44 0x0000000000000000 in ?? ()
    #45 0x0000000000000000 in ?? ()
    #46 0x0000000000000000 in ?? ()
    #47 0x0000000000000000 in ?? ()
    #48 0x0000000000000000 in ?? ()
    #49 0x0000000000000000 in ?? ()
    #50 0x0000000000000000 in ?? ()
    #51 0x0000000000000000 in ?? ()
    #52 0x0000000000000000 in ?? ()
    #53 0x0000000000000000 in ?? ()
    #54 0x0000000000000000 in ?? ()
    #55 0x0000000000000000 in ?? ()
    #56 0x0000000000000000 in ?? ()
    #57 0x0000000000000000 in ?? ()
    #58 0x0000000000000000 in ?? ()
    #59 0x0000000000000000 in ?? ()
    #60 0x0000000000000000 in ?? ()
    #61 0x0000000000000000 in ?? ()
    #62 0x0000000000000000 in ?? ()
    #63 0x0000000000000000 in ?? ()
    #64 0x0000000000000000 in ?? ()
    #65 0x0000000000000000 in ?? ()
    #66 0xffffffff8117f7c8 in sleepq_chains ()
    #67 0xfffffe0006460888 in ?? ()
    #68 0x0000000000000000 in ?? ()
    #69 0xfffffe0006460460 in ?? ()
    #70 0xffffff81f0f57b00 in ?? ()
    #71 0xffffff81f0f57aa8 in ?? ()
    #72 0xfffffe00062a2460 in ?? ()
    #73 0xffffffff8085dda2 in sched_switch (td=0xffffffff80878420, 
        newtd=0xffffff80012e6748, flags=Variable "flags" is not available.
    ) at /usr/src/sys/kern/sched_ule.c:1848
    Previous frame inner to this frame (corrupt stack?)
    (kgdb)
    

    config I use :  ( i am using about 100 or more VLAN )

    default:
     load s0
     load s1
     load s2
     load s3
    s0:
     new -i ng0 s0 ppplink0
     load generic
    s1:
     new -i ng1 s1 ppplink1
     load generic
    s2:
     new -i ng2 s2 ppplink2
     load generic
    s3:
     new -i ng3 s3 ppplink3
     load generic
    
    generic:
    # Multihomed multilink PPPoE server
    #
    # Create clonable bundle template
        create bundle template B
        set ippool add pool1 172.16.0.3 172.16.255.255
    # Set IP addresses. Peer address will be later replaced by RADIUS.
        set ipcp ranges 172.16.0.2 ippool pool1
        set ipcp dns 4.2.2.4 8.8.8.8
    # Create link template with common info
        create link template common pppoe
    # Enable multilink protocol
        set link enable multilink
    # Set bundle template to use
        set link action bundle B
    # Enable peer authentication
        set link disable chap eap
        set link enable pap
        set pppoe service "*"
    

    Tunings :

    kern.ipc.maxpipekva=536870912
    net.inet.tcp.delayed_ack=0
    net.isr.maxthreads=7
    net.isr.direct=1
    net.isr.direct_force=1
    net.isr.bindthreads=0
    kern.maxfiles=204800
    kern.maxfilesperproc=200000
    net.graph.maxalloc=4096
    kern.maxusers=2048
    kern.ipc.maxsockbuf=16777216
    kern.ipc.nmbclusters=262144
    kern.ipc.somaxconn=32768
    kern.ipc.maxsockets=204800
    

    where is the problem ! or what am i doing wrong ? :(

     
  • changizy
    changizy
    2012-09-02

    sorry wrong dump core! that one is for when i use nat with IPFW !
    here it is the punic i get when dont use IPFW NAT or any other NAT ! :D

    PPPOESERVER dumped core - see /var/crash/vmcore.0
    Wed Aug 29 15:16:52 IRDT 2012
    FreeBSD PPPOESERVER 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Wed Aug 29 13:56:46 IRDT 2012     root@pppoe_server:/usr/obj/usr/src/sys/GENERIC  amd64
    panic: page fault
    GNU gdb 6.1.1 [FreeBSD]
    Copyright 2004 Free Software Foundation, Inc.
    GDB is free software, covered by the GNU General Public License, and you are
    welcome to change it and/or distribute copies of it under certain conditions.
    Type "show copying" to see the conditions.
    There is absolutely no warranty for GDB.  Type "show warranty" for details.
    This GDB was configured as "amd64-marcel-freebsd"...
    Unread portion of the kernel message buffer:
    Fatal trap 12: page fault while in kernel mode
    cpuid = 0; apic id = 00
    fault virtual address   = 0x5b600000099
    fault code      = supervisor read data, page not present
    instruction pointer = 0x20:0xffffffff8093738c
    stack pointer           = 0x28:0xffffff824d18c6d0
    frame pointer           = 0x28:0xffffff824d18c700
    code segment        = base 0x0, limit 0xfffff, type 0x1b
                = DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags    = interrupt enabled, resume, IOPL = 0
    current process     = 0 (em0 taskq)
    trap number     = 12
    panic: page fault
    cpuid = 0
    KDB: stack backtrace:
    #0 0xffffffff8086b1de at kdb_backtrace+0x5e
    #1 0xffffffff80835d97 at panic+0x187
    #2 0xffffffff80b40ec0 at trap_fatal+0x290
    #3 0xffffffff80b41209 at trap_pfault+0x1f9
    #4 0xffffffff80b416cf at trap+0x3df
    #5 0xffffffff80b2bbff at calltrap+0x8
    #6 0xffffffff8093d8d8 at ng_pppoe_rcvdata_ether+0x2b8
    #7 0xffffffff80938fdb at ng_apply_item+0x22b
    #8 0xffffffff80937f8e at ng_snd_item+0x39e
    #9 0xffffffff808ea897 at ether_demux+0x127
    #10 0xffffffff808eab94 at ether_nh_input+0x1f4
    #11 0xffffffff808f4e8b at netisr_dispatch_src+0x20b
    #12 0xffffffff808ea7df at ether_demux+0x6f
    #13 0xffffffff808eab94 at ether_nh_input+0x1f4
    #14 0xffffffff808f4e8b at netisr_dispatch_src+0x20b
    #15 0xffffffff8047303a at em_rxeof+0x1ca
    #16 0xffffffff804734bb at em_handle_que+0x5b
    #17 0xffffffff808774e5 at taskqueue_run_locked+0x85
    Uptime: 23m37s
    Dumping 662 out of 8173 MB:..3%..13%..22%..32%..42%..51%..61%..71%..83%..92%
    Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_socket.ko
    Reading symbols from /boot/kernel/ng_mppc.ko...Reading symbols from /boot/kernel/ng_mppc.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_mppc.ko
    Reading symbols from /boot/kernel/rc4.ko...Reading symbols from /boot/kernel/rc4.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/rc4.ko
    Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_ether.ko
    Reading symbols from /boot/kernel/ng_tee.ko...Reading symbols from /boot/kernel/ng_tee.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_tee.ko
    Reading symbols from /boot/kernel/ng_iface.ko...Reading symbols from /boot/kernel/ng_iface.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_iface.ko
    Reading symbols from /boot/kernel/ng_ppp.ko...Reading symbols from /boot/kernel/ng_ppp.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_ppp.ko
    Reading symbols from /boot/kernel/ng_vjc.ko...Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done.
    done.
    Loaded symbols for /boot/kernel/ng_vjc.ko
    #0  doadump (textdump=Variable "textdump" is not available.
    ) at pcpu.h:224
    224 pcpu.h: No such file or directory.
        in pcpu.h
    (kgdb) #0  doadump (textdump=Variable "textdump" is not available.
    ) at pcpu.h:224
    #1  0xffffffff808358d5 in kern_reboot (howto=260)
        at /usr/src/sys/kern/kern_shutdown.c:442
    #2  0xffffffff80835d81 in panic (fmt=Variable "fmt" is not available.
    )
        at /usr/src/sys/kern/kern_shutdown.c:607
    #3  0xffffffff80b40ec0 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
    )
        at /usr/src/sys/amd64/amd64/trap.c:819
    #4  0xffffffff80b41209 in trap_pfault (frame=0xffffff824d18c620, usermode=0)
        at /usr/src/sys/amd64/amd64/trap.c:735
    #5  0xffffffff80b416cf in trap (frame=0xffffff824d18c620)
        at /usr/src/sys/amd64/amd64/trap.c:474
    #6  0xffffffff80b2bbff in calltrap ()
        at /usr/src/sys/amd64/amd64/exception.S:228
    #7  0xffffffff8093738c in ng_address_hook (here=0x0, item=0xfffffe0199acab00, 
        hook=0xfffffe01b7a5ac00, retaddr=0)
        at /usr/src/sys/netgraph/ng_base.c:3487
    #8  0xffffffff8093d8d8 in ng_pppoe_rcvdata_ether (hook=Variable "hook" is not available.
    )
        at /usr/src/sys/netgraph/ng_pppoe.c:1655
    #9  0xffffffff80938fdb in ng_apply_item (node=0xfffffe0024cb9900, 
        item=0xfffffe0199acab00, rw=0) at /usr/src/sys/netgraph/ng_base.c:2318
    #10 0xffffffff80937f8e in ng_snd_item (item=Variable "item" is not available.
    )
        at /usr/src/sys/netgraph/ng_base.c:2235
    #11 0xffffffff808ea897 in ether_demux (ifp=0xfffffe002400b800, m=Variable "m" is not available.
    )
        at /usr/src/sys/net/if_ethersubr.c:954
    #12 0xffffffff808eab94 in ether_nh_input (m=Variable "m" is not available.
    )
        at /usr/src/sys/net/if_ethersubr.c:756
    #13 0xffffffff808f4e8b in netisr_dispatch_src (proto=9, source=Variable "source" is not available.
    )
        at /usr/src/sys/net/netisr.c:1013
    #14 0xffffffff808ea7df in ether_demux (ifp=0xfffffe00054aa000, 
        m=0xfffffe0199e12600) at /usr/src/sys/net/if_ethersubr.c:846
    #15 0xffffffff808eab94 in ether_nh_input (m=Variable "m" is not available.
    )
        at /usr/src/sys/net/if_ethersubr.c:756
    #16 0xffffffff808f4e8b in netisr_dispatch_src (proto=9, source=Variable "source" is not available.
    )
        at /usr/src/sys/net/netisr.c:1013
    #17 0xffffffff8047303a in em_rxeof (rxr=0xfffffe0009044400, count=97, 
        done=0x0) at /usr/src/sys/dev/e1000/if_em.c:4340
    #18 0xffffffff804734bb in em_handle_que (context=Variable "context" is not available.
    )
        at /usr/src/sys/dev/e1000/if_em.c:1518
    #19 0xffffffff808774e5 in taskqueue_run_locked (queue=0xfffffe0009042200)
        at /usr/src/sys/kern/subr_taskqueue.c:308
    #20 0xffffffff80878466 in taskqueue_thread_loop (arg=Variable "arg" is not available.
    )
        at /usr/src/sys/kern/subr_taskqueue.c:497
    #21 0xffffffff8080990f in fork_exit (
        callout=0xffffffff80878420 <taskqueue_thread_loop>, 
        arg=0xffffff800230c748, frame=0xffffff824d18cc50)
        at /usr/src/sys/kern/kern_fork.c:995
    #22 0xffffffff80b2c12e in fork_trampoline ()
        at /usr/src/sys/amd64/amd64/exception.S:602
    #23 0x0000000000000000 in ?? ()
    #24 0x0000000000000000 in ?? ()
    #25 0x0000000000000000 in ?? ()
    #26 0x0000000000000000 in ?? ()
    #27 0x0000000000000000 in ?? ()
    #28 0x0000000000000000 in ?? ()
    #29 0x0000000000000000 in ?? ()
    #30 0x0000000000000000 in ?? ()
    #31 0x0000000000000000 in ?? ()
    #32 0x0000000000000000 in ?? ()
    #33 0x0000000000000000 in ?? ()
    #34 0x0000000000000000 in ?? ()
    #35 0x0000000000000000 in ?? ()
    #36 0x0000000000000000 in ?? ()
    #37 0x0000000000000000 in ?? ()
    #38 0x0000000000000000 in ?? ()
    #39 0x0000000000000000 in ?? ()
    #40 0x0000000000000000 in ?? ()
    #41 0x0000000000000000 in ?? ()
    #42 0x0000000000000000 in ?? ()
    #43 0x0000000000000000 in ?? ()
    #44 0x0000000000000000 in ?? ()
    #45 0x0000000000000000 in ?? ()
    #46 0x0000000000000000 in ?? ()
    #47 0xffffffff8117ed78 in sleepq_chains ()
    #48 0xfffffe0005472428 in ?? ()
    #49 0x0000000000000000 in ?? ()
    #50 0xfffffe0005472000 in ?? ()
    #51 0xffffff824d18cb00 in ?? ()
    #52 0xffffff824d18caa8 in ?? ()
    #53 0xfffffe00052ab460 in ?? ()
    #54 0xffffffff8085dda2 in sched_switch (td=0xffffffff80878420, 
        newtd=0xffffff800230c748, flags=Variable "flags" is not available.
    ) at /usr/src/sys/kern/sched_ule.c:1848
    Previous frame inner to this frame (corrupt stack?)
    (kgdb)
    
     
  • What you see in /var/log/messages and console ?
    Or you may enable KTR and option in you custom kernel and see, what messages passed to netgraph

     
  • changizy
    changizy
    2012-09-07

    in /var/log/messages before punic :

    Aug 29 15:05:21 PPPOESERVER kernel: nd6_setmtu0: new link MTU on ng5455 (512) is too small for IPv6
    Aug 29 15:16:28 PPPOESERVER syslogd: kernel boot file is /boot/kernel/kernel
    Aug 29 15:16:28 PPPOESERVER kernel: O_LINKLOCAL>
    Aug 29 15:16:28 PPPOESERVER savecore: reboot after panic: page fault
    Aug 29 15:16:28 PPPOESERVER savecore: writing core to vmcore.0
    Aug 29 15:45:22 PPPOESERVER kernel: WARNING: attempt to domain_add(netgraph) after domainfinalize()

     
  • No.
    In you first message I see these lines:
    #5 0xffffffff80b2bbff at calltrap+0x8
    #6 0xffffffff8093d8d8 at ng_pppoe_rcvdata_ether+0x2b8
    #7 0xffffffff80938fdb at ng_apply_item+0x22b
    #8 0xffffffff80937f8e at ng_snd_item+0x39e

    In all this functions you can see line like
    CTR6(KTR_NET, "%20s: bla-bla…

    This are KRACE and KTR messages
    You can be enable their features by include KTRACE and KTR options in you custom kernel